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SOME ASPECTS OF THE 



EDUCATIONAL CHANGE DYNAMIC: 
SETTING SCHOOL AUTONOMY 
AND EVALUATION IN CONTEXT 

Cecilia Braslavsky 

The construction of modern schools and national education systems became possi- 
ble through two models of management and educational policy. The first was the 
‘community’ model and the second was the ‘nation-State’ model. Both can be found 
throughout developed countries, sometimes in open conflict, at other times in more 
or less productive partnership. 

The ‘community’ approach was the guiding principle in the Anglo-Saxon world. 
In England and in the Netherlands, for instance, schools were set up by parish church- 
es in close collaboration with the local community. As municipalities were created, 
they set up lay establishments in the most prosperous towns of Northern Europe 
and also in the north of Italy and even France in the sixteenth century, in competi- 
tion with ecclesiastical institutions. This approach was even more evident in the 
wave of setting up and expanding education in North America. Adopting modern 
terminology, we could say that these were based on the ‘bottom-up’ approach. 

The ‘nation-State’ model was most popular from the eighteenth century in the 
major States, first in those of the feudal type or the later ones which benefited from 
State support for capitalist and industrial modernization, such as the despotic States 
of Spain and France or the French republics of the eighteenth to twentieth centuries. 
Once again adopting the terminology of professional jargon, we may say that these 
are examples of the ‘top-down’ approach. 

The ‘communal’ model was guided by a decision-making process that today 
we would classify as ‘micro-political’. Schools took an enormous number of deci- 
sions concerning what and how to teach. They decided, for example, syllabi and 
curricula without much interference from the public authorities. With a large amount 
of power entrusted to the school, the interactions between the different partners 
were — ^and still are — very important. We may even say that they were delegated this 
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power by higher levels of administration. Using this model, throughout the centuries 
communities were slowly building a strong civil society which organized different 
local communities through exchanges, dialogue and debate. 

Inspired by the success of the development of the ‘community’ model into 
strong civil societies with States at their service and — sometimes — also critical remarks 
about the undesirable effects of the ‘nation-State’ model in limiting teachers’ peda- 
gogical freedom, a large number of thinkers and institutions put forward the idea 
in the 1980s and 1990s of strengthening the autonomy of the school as a way of 
overcoming educational problems. This proposal to strengthen the autonomy of the 
school was accompanied by similar proposals to strengthen the mechanisms for eval- 
uating the outcomes of pupils’ learning. The Open File in this issue of PROSPECTS 
presents the school autonomy/evaluating-the-outcomes-of-pupil-learning equation. 

Probably one of the principal lessons learned from these two decades was that 
in the search for alternatives it is necessary to combine components of bottom-up 
management (for example, by strengthening school autonomy) with others from the 
top-down approach (such as certain types of standardized evaluation, in order to 
allow different levels of learning to be compared). 

Even so, the moment has probably come to avoid falling into the trap of a dia- 
logue or discussion over the advantages between the ‘top-down’ and the ‘bottom-up’ 
models. In fact, we now have technical means available that allow us to construct 
a network model taking advantage of new technologies. It should operate by bring- 
ing together various levels of management in a much more intense interaction and 
redefining them as co-operative partners (and not in a hierarchical relationship) in 
processes through which all existing institutions intervene in an articulated and pro- 
ductive manner. The question that must be asked is what are the competences required 
to create such relationships, whether the partners have them or not and — if they 
don’t have them — how do they get them. 

In this connection, it is important to take into account the characteristics of 
each context. The two articles that we are printing in this issue of PROSPECTS on 
education in Afghanistan link its past and its present, and provide evidence that no 
abstract theory about management models is applicable and pertinent in all situa- 
tions. There may be general guidelines which direct our thinking at one stage, but 
they must always be interpreted and revised in each case according to circumstances. 
How do we begin the educational reconstruction of Afghanistan? What do we learn 
from its past, its present, and the efforts to combine autonomy and evaluation in 
other parts of the world? These and other questions could help those concerned by 
this situation towards finding the answers they are looking for. 

This issue of PROSPECTS closes with a profile of Basil Bernstein, who was 
recognized as a leading sociologist throughout the world. His pioneering work over 
the past four decades illuminated our understanding of the relationship among polit- 
ical economy, family, language and schooling. While his early works explored the 
relationships among the social division of labour, the family and the school, his later 
works on pedagogic discourse, practice and educational transmissions led to a the- 
ory of social and educational codes and their effect on social reproduction. 
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Evaluation is a central concern in education policy and education research today.* 
The interest in it is not new. Education systems are very much concerned by the 
‘demand’ for assessment, which for many years focused on the activity of pupils 
(their learning) and, to a lesser extent, that of teachers (their teaching skills). However, 
for some twenty years now the field has been evolving in two ways: on the one hand, 
evaluation has moved beyond school circles to become a front-line political issue; and 
on the other, it has been extended from pupils and teachers — let us say, people taken 
individually — to cover schools, systems and training policies. 
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The institutional effect 

With regard to schools, the actual concept of an ‘institutional effect’ and its exis- 
tence were not at first recognized. At the end of the 1960s, James Coleman^ and 
Christopher Jencks^ questioned whether it was possible to alter the distribution 
of academic performance by acting on school-specific variables, such as class size, 
teaching hours, teachers’ qualifications and the organization of the school. 
According to Jencks: ‘Differences between schools have rather trivial long-term 
effects, and [...] eliminating differences between schools would do almost noth- 
ing to make adults more equal. Even eliminating differences in the amount of 
schooling people get would do relatively little to make adults more equal’. ^ These 
conclusions, which minimized the impact of differences between schools on per- 
formance, were not accepted unanimously and were the starting point for a long 
series of studies which opened up the ‘black box’ of the school. Researchers went 
into schools not to assess pupils, but to see what really went on inside, to observe 
the way in which the school functioned and to discover factors which did influ- 
ence pupil attainment. This entry into the ‘black box’ enabled a number of vari- 
ables to be pinpointed which can be said to have a definite and positive effect on 
learning. 

The study by Michele Rutter and her fellow researchers, B. Morgan, P. Mortimer 
and G. Houston, on the effects of secondary schools in England,^ which was among 
the critical works produced in reaction to the work of Coleman and Jencks, was an 
important step forward and marked a necessary shift in the history of school eval- 
uation. The British researchers demonstrated that the effects of schools on pupil 
attainment are far from negligible. In so doing they opened up a new field of study, 
generating a whole flurry of research throughout the 1980s, notably on ‘school qual- 
ity and development’^ and effective schools. OECD’s International School 
Improvement Project (ISIP)^ was part of the new trend and was to contribute sig- 
nificantly to its development. 

Editorial series on best school practice are one of the unexpected by-products 
of these studies on the quality and effectiveness of schools. They were produced in 
order to ‘spread the good news’, that is, to inform practitioners of solutions that, 
on the basis of research findings, could be said to improve education. The explana- 
tory brochure What works^ was one of the very first examples. The idea was taken 
up again a few years later by CERI, the OECD Centre for Educational Research and 
Innovation, which in the 1990s started to publish reports in a series entitled 
‘Innovations that work’. At the end of the 1990s, the International Bureau of 
Education (IBE) carried on this work by publishing its series ‘Educational practices’ 
in collaboration with the International Academy of Education.^ 

These thirty years of research into schools have resulted in a considerable mass 
of data and analyses which have focused attention on the functioning of schools by 
providing information that has itself led to changes in how the results of teaching 
and the ways in which pupils learn are interpreted. As Ballion summed it up: 
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research work on the ‘effective school’ reveals that at this level [that of the school] it is pos- 
sible not only to identify differential factors of effectiveness, explaining for instance that 
schools with identical contextual and compositional variables can have unequal results, and 
that in any one school, one primary school, there can be, as a recent French study has shown, 
teachers who are effective and others who are not; but also that the school, even if it is not 
the only one, is an appropriate level of action for improvement. 

Schools have therefore ceased to be a mysterious world, shielded from indiscreet 
and prying eyes, in which teachers tend to shut themselves off in order to defend 
their mission. The research has also transformed the way in which schools are pre- 
sented, not as ‘production units’, but as educational communities, learning 
organizations where professional teams are at work. 

From knowing about schools to evaluating them 

The debate on evaluating schools has a relatively short history, as becomes clear, 
for instance, on reading the report of the working group on the evaluation of the 
pre-service training system written by Robert Ballion as part of the work of the edu- 
cation/training/research commission of the tenth policy guidance plan in France, 
published in 1991.’’ Reference to school evaluation in the report is confined to a 
single paragraph entitled ‘The evaluation of production units’. The school is pre- 
sented as a production unit, that is, as an ‘entity generating its own functionality’; 
it can therefore no longer be conceived of ‘as simply an aggregate of individuals’,’^ 
it is effective in its own right. School evaluation is not addressed in The interna- 
tional encyclopaedia of educational evaluation^ published in 1990.’^ 

Nevertheless, the new way of looking at the school has introduced the idea 
that it is just as important to evaluate education systems themselves, by evaluating 
the performance of the pupils taken as a whole and the quality of schools, as it is 
to evaluate individual performance (pupils and teachers). Moreover, this evaluation 
in the ‘quality and development of schools’ research context aims above all to be 
‘formative’; it is not a question of ranking or sanctioning schools, but of providing 
professionals with the tools to reflect on their practices. As J.-L. Derouet has point- 
ed out, school evaluations are useful as a potential means of improving life at school, 
for instance ‘as a way of bringing about a measure of civil peace within an institu- 
tion riven by antagonism between clans’, or an instrument ‘of cultural integration 
for a community’.’^ Evaluation can alert the teaching staff to where its knowledge 
is lacking, inform it of what is basically needed for the smooth running of the school, 
and so on. 

School evaluation: fears and risks 

From the early 1990s the success of educational indicators aroused even further 
interest in these challenges, but at the same time introduced into the field of research 
and policy a new dimension which had previously been approached only with great 
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caution, namely comparison between schools, variations within schools and between 
schools, and the difference in results between education systems. To that was added 
the fact that in many countries school evaluation was imposed as part of a reform 
package giving schools more independence. The implication of these reforms'^ — 
freedom of choice of schools, training vouchers, liberalization of methods of 
administration of school systems — increased the need for knowledge about the 
nature, performance and functioning of schools, but in so doing gave evaluation, at 
least initially, a punitive function as much as an improving one. 

This development has raised both legitimate and exaggerated expectations and 
fears about evaluation. While, on the whole, there is agreement on the need to eval- 
uate education and not just pupils taken individually, there is significant disagreement 
as to the nature and modalities of the procedures to be adopted. The proliferation 
of experimental and formal evaluation programmes has clarified the debate only in 
part. Scientists are divided on the subject of the methods and effect of evaluation. 
As to teachers, they remain suspicious and are unconvinced by new schemes or pro- 
posals for participatory evaluation, arguments extolling the formative nature of 
evaluation and demonstrations of the advantages and merits of self-evaluation as 
compared with any form of external evaluation. Evaluation smacks of heresy. In 
countries where there is a wide variety of pupil assessment practices, such as in the 
United States of America, most teachers have to submit routinely to the chore of 
tests and questionnaires which, it may be supposed, a substantial majority would 
willingly do without. 

School evaluation is therefore being introduced in a climate marked by dis- 
trust, doubt and scepticism. The purpose of this Open File. is to take stock of 
developments in this field. To some extent it is complementary to the open file on 
the evaluation of education systems published in vol. XXVIII, no. 1, March 1998, 
of Prospects^ which in a way foreshadowed it.'^ 

Interesting avenues for research are opening up and some of them are explored 
in the articles in this Open File. Having said that, we must be under no illusion: as 
pointed out above, the evaluation of schools can also be used to regulate the edu- 
cation market and stimulate competition between schools. The texts presented here 
do not lend themselves to this purpose and even draw attention to this kind of mis- 
use which jeopardizes the transformation of schools into privileged places of learning 
and development for pupils. 

The Open File 

The Open File we present here contains two types of text: (a) articles based on 
European situations, describing ways of evaluating schools that are consistent with 
their independence; and (b) papers illustrating how the independence of schools is 
coupled with their evaluation in four different national settings. 

Needless to say, any operation to evaluate schools must, in order to succeed, 
be acceptable to teachers, whatever the type and method of evaluation adopted. The 
panoply of evaluation methods is such today that any may be envisaged as long as 
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its scope, limits, risks, advantages and disadvantages are fully grasped. There is nev- 
ertheless one factor that can never be overlooked if the evaluation is expected to 
help improve education and learning, and that is the trust of teachers. If they are 
not convinced of the benefits that they, in the exercise of their profession, can derive 
from evaluation, there is a potential risk of wasting resources and time, producing 
inaccurate data and sending out the wrong signals. Evaluation of schools cannot 
therefore be carried out without the teachers. It is important to take the full mea- 
sure of this elementary axiom in the knowledge that the conviction of teachers in 
this regard can by no means be taken for granted and that in general evaluation, 
despite all the advances made, in particular in recent decades, is still not an integral 
part of educational practices or, even more so, of school management. All the arti- 
cles in the file hinge on this requirement: how do we reconcile evaluation mechanisms 
with teachers’ concerns? How do we design evaluations that are acceptable to the 
key players in education? What happens when evaluation practices that disregard 
the teachers’ expectations and ignore their daily professional concerns are forced 
on schools? To what experiences may we refer in order to appraise factors of suc- 
cess and failure in evaluation procedures? Is there a strong connection between 
typologies of evaluation and teacher behaviour? 

The article by Abi-Saab and Alt, the first in the European part of the file, 
describes France’s experience with secondary school indicators (IPES). Among the 
cases described in this file, this is the one that proposes the closest linkage between 
a source outside the schools and the schools themselves. School-specific data, processed 
statistically and reduced to a set of key indicators, are supplied to the schools, togeth- 
er with a special computer application, so that they can be used as benchmarks to 
assist teachers and school heads in analysing outcomes. 

In the same vein, the text by Saunders presents studies carried out in the United 
Kingdom to measure the value added to schools, in other words to correct and rel- 
ativize the raw evaluation data by incorporating into them the particular variables 
of a school. This approach meets a demand for fairness, as it is not possible to put 
on an equal footing schools which are obviously very different from each other in 
a number of ways. Teaching in a school in a disadvantaged area is not the same as 
teaching in a school in a prosperous area: the pupils have a different cultural and 
social capital, the schools do not enjoy the same facilities and the staff itself varies 
considerably from one school to the next. These parameters must, then, be taken 
into account when a school is being evaluated. Saunders sums up the characteris- 
tics of the value-added-based approach and makes recommendations drawn from 
practical experience regarding the measurement and use of added value. In addi- 
tion, she shows the way in which schools make use of the data on their added value 
that are measured by specialist centres and made available to them. The French and 
British experiences, though inspired by different theories, resemble each other in 
that they are characterized by the transmission to schools of tailored statistical infor- 
mation which should or could provide input for self-evaluation. 

The experiences in the Netherlands and in Geneva described in the papers by 
Hendriks et al. and Favre, mark an important new stage, that of the internalization 
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of evaluation practices and the disappearance from the scene of specialist external 
bodies providing ad hoc data. This can be seen in the ZEBO project, which endeav- 
ours to provide the self-evaluation approach with credible and valid instruments. 
In other words, it is not enough to carry out self-evaluation; it must be scientifical- 
ly sound, leaving no room for improvization and approximation. The question then 
is knowing how to equip schools to be fully in command of the evaluation opera- 
tion by making use of the potential that independence can free up within school 
communities. The solution consists in ensuring that the school itself has control over 
the evaluation instruments and can identify with their purpose and use. In the Dutch 
experience, it is the stakeholders themselves — administration, teachers and pupils — 
who decide what information to gather, who gathers it and who processes it. 

The Geneva study described by Favre falls within the context of a major reform 
of primary education in the Canton of Geneva. However, it concerns a phase that 
is upstream of evaluation and self-evaluation practices. This means that the instru- 
mental dimension of the analysis is lost but in exchange it has a density and range 
that were missing altogether from previous exercises, since it covers the history of 
the school and an analysis of its organization and the elements that go into making 
it an educational community. The object is to describe the particular organization- 
al rationale into which evaluation and self-evaluation necessarily fit. This rationale 
helps to explain, at least in part, teacher resistance to evaluation and any departures 
from the original purpose of the evaluation. As a result, the Geneva study may open 
up another field of study, namely the relationship between organizational learning 
and school evaluation and self-evaluation practices, or the ways in which the stake- 
holders’ own knowledge of their school as an organization can be used for purposes 
of evaluation. 

In the second part of the file there are four studies describing national policies 
dealing with the problem of regulating the independence of schools. The situations 
described in this section are very different. Fiske and Ladd shed light on New 
Zealand’s experience in the context of an ambitious reform launched since 1989 
and aimed at decentralizing the administration of the education system, giving con- 
siderable independence to schools, but without setting up a restrictive evaluation 
and self-evaluation mechanism to offset the effects of decentralization and act as an 
instrument to regulate the system as a whole. 

Jansen describes developments in South Africa, where the requirement for 
school evaluation has taken the form of officially requiring schools to carry out their 
own evaluation, the object being to regulate schools and ensure that they align them- 
selves on the objectives of national education policy. The independence of schools 
and the concern to embed and strengthen good practices within schools that led 
them to obtain increasingly satisfactory results are overshadowed by the imposition, 
in an almost underhand way, of a sophisticated control mechanism entailing the 
hijacking of self-evaluation. It is legitimate in this case to talk of hijacking. The inde- 
pendence of schools is to all intents and purposes devoid of meaning and scope. 

The experience of Chile described by Casassus provides insights into an inter- 
esting new approach to school evaluation which aims to reduce teachers’ distrust 
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of evaluation schemes and secure their commitment to a process which they can 
only partly call their own in order to ensure that, in the end, evaluation represents 
an opportunity to improve education. The Chilean approach could be said to be 
halfway between the impasse of the South African model analysed by Jansen and 
the extreme solution of a dilution of self-evaluation in a self-referential practice that 
would be found in the Geneva model described by Favre, if it were seen as a form 
of self-evaluation. The school evaluation system introduced in Chile is not centred 
on teachers but on the school as a recognized community through incitements and 
rewards provided for and paid when the school is managed effectively by the teach- 
ing team. 

Lastly, the paper by Schmelkes gives a particularly clear illustration of the dif- 
ficulties and deadlock encountered when seeking to reconcile the independence of 
schools with evaluation, as we have already observed in South African education 
policy. The balance between the independence of schools and the evaluation of 
education is a delicate one; it is to be found neither in the external nor in the inter- 
nal evaluation approach. The New Zealand solution puts a premium on 
independence but does not provide an evaluation model that takes account of the 
need to move a large group of schools forward, together if possible, in order to 
avoid the exacerbation of disparities in the education on offer. In France, the 
Netherlands and the United Kingdom ways and means of combining complex eval- 
uation mechanisms for the benefit of schools are being tested. In Mexico, in a 
situation which is far less advantageous economically, the independence of schools 
is a theme that policies address with caution — there are reforms along those lines, 
as in New Zealand and Chile, but in the form of closely-controlled experiments, 
warranted by the enormous diversity in the supply of education available in the 
country. The fear that independence might lead to greater, rather than less, injus- 
tice in education is in fact a basic theme in this debate: we still do not have proof 
that independence — even when it is implemented partially or selectively — makes 
for a better distribution of educational opportunity, but equally we know that there 
is no definite proof that the model of centralized management established to ensure 
the uniform and therefore fair development of education throughout a country has 
been successful either. In the second case, there is abundant data, even though in 
many countries — not only in Mexico — the authorities disregard them or do not 
even know that they exist. 

In conclusion, even if on a theoretical level it is acknowledged that the self- 
evaluation of schools has not only a formative value but should be part of the raft 
of new forms of regulation of education systems, in practice, the coexistence — 
although it would be more accurate to say intermeshing — of independence and 
evaluation has proved highly problematic. However, as shown by the experiences 
presented in this Open File, the intermeshing of central and local regulation is pos- 
sible, but cannot be done on the cheap: it requires considerable investment in terms 
of scientific and financial input, but above all in terms of clearly formulated objec- 
tives that enable schools to know from the outset what to expect, without excessive 
illusions but also without suspicion. 
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Each year since 1994, the French Ministry of Education has provided the heads of 
public secondary schools, through the relevant academie^^ with a set of indicators 
to facilitate the running of schools, known as IPES (Indicateurs pour le pilotage des 
etablissements secondaires), together with data for each academie and for France 
as a whole and computer software supplied by the ministry. 

This set of indicators satisfies a number of needs felt by those who work in the 
education system. After describing these needs and the chosen way of meeting them, 
this article will focus on assessment of the ‘performance’ of lycees (upper secondary 
schools), through the concept of added value in particular, and will go on to exam- 
ine the conditions necessary for the optimal use of these indicators. 
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The origin and organization of the IPES system 

THE NEED TO MEET SEVERAL DIFFERENT TYPES OF REQUIREMENT 



The concern to provide schools with indicators dates back to the 1970s. There has, 
however, been a change in the objectives and conception of these indicators in the 
last ten years, due partly to changes in school management policy and a strong 
demand for assessment indicators to facilitate measurement of a school’s perfor- 
mance, but also to the evolution of information systems. 

In the 1970s and until the end of the 1980s, for example, school ‘management 
charts’ (tableaux de bord)^ developed at the ministry, contained indicators that facil- 
itated analysis and decision-making. They proved, however, to be a complex and 
cumbersome tool because of the difficulty of obtaining the necessary information. 

The rationale underlying the school plans (projets d'etablissement)^ an idea 
introduced in' the early 1980s and consolidated by the 1989 education act, prompt- 
ed the design of new indicators to assist initial diagnosis, the making of choices, the 
fixing of objectives and the evaluation of the plan’s implementation. Indicators were 
developed locally by the various parties involved in the system, with the result that 
the plans could not be assessed in a homogeneous manner. 

Furthermore, the publication by the press of gross baccalaureat pass rates and 
the lycees ‘league tables’ based solely on these rates, ^ ignoring various other aspects 
of their activities and their context, combined with the popularity of these publica- 
tions with the general public, made it urgent for the education system to produce 
its own reports on the performance of schools by providing indicators of success 
that took account of the problems they encountered. As a result, three performance 
indicators for lycees, based on the results of their pupils in the baccalaureat exam- 
ination and included among the battery of IPES indicators, were published and made 
available to all those involved in the education system, from senior officials to par- 
ents.^ 

This was also a time of rising unemployment, especially for those with fewer 
qualifications, and of increased job insecurity and a general malaise prompted by 
these trends. Scholastic success and obtaining the highest possible qualifications thus 
came to be regarded as a safeguard against this anxiety about the future. 

These factors — the many forms of assessment, the ranking of lycees by certain 
newspapers or periodicals, the introduction of ‘school plans’, national and region- 
al investment in education, and public disquiet about the future — contributed to the 
rise of a demand for indicators for the evaluation of schools. At the same time, 
progress in the procedures used for the collection of data on pupils, teachers and 
educational structures has made it possible, using new, computerized information 
systems, to standardize the information gathered and thus provide schools with a 
standard set of indicators. 
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Organization of the system: 
several types of indicators 

DESCRIPTION AND CONTENT OF THE IPES SYSTEM 

The IPES system makes no claim to cover all areas necessary for a fully compre- 
hensive knowledge and an all-embracing assessment of each school. It offers an 
approach to certain fundamental factors that facilitate an initial appraisal of the sit- 
uation of the school but does not permit an exhaustive evaluation. 

The areas covered are: characteristics of the population, operation of the school, 
resources, the local context and results. 

Characteristics of the student population 

These indicators apply to: 

• socio-demographic characteristics, i.e. the percentage of disadvantaged pupils 
in the school as a whole and in each year, the percentage of pupils over 18 , the 
percentage of foreign pupils; 

• educational characteristics and previous schooling, i.e. the percentage of pupils 
repeating a year, the percentage of pupils two or more years behind, the edu- 
cational standard of pupils entering sixieme^ the first year of upper secondary 
general and technical education^ and of upper secondary vocational education, 
the percentage of pupils coming from the same school, and the percentage of 
pupils coming from private schools. 

Operational indicators 

These indicators attempt to give an objective picture of the way schools operate by 
showing the resources available to each school and taking into account its economic 
and social environment. 

The resources available to the school: In order to gauge and describe the resources 
of each school, the IPES includes indicators concerning the hours worked, the main 
courses of study, the languages taught and the teachers. 

The hours worked by a school are measured by two indicators, one for the 
number of hours of teaching in relation to the number of pupils and the other for 
the overall number of hours, including hours both of teaching, as shown by the 
timetable, and of back-up activities, i.e. the ‘other hours’ spent by all adult persons 
working in contact with the pupils — heads and senior staff, counsellors and career 
officers, supervisors of practical work and their assistants, documentalists, foreign- 
language assistants, monitors, housemasters, and so forth. 

The indicators for the main courses of study and on the languages taught pro- 
vide an overview of the range of subjects and languages offered by the school. In 
addition, they give the take-up rate in each case. 
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The indicators for teachers show their age, sex and level of academic qualifi- 
cation by the different categories of staff. 

Data of a more qualitative nature: These data provide information on qualitative 
aspects such as the quality of life in the school, ancillary services and the status of 
lycee students, or factors indicative of the school climate, such as the accidents that 
occur and the proportion of timetable hours when teaching is actually provided to 
a student. These last two indicators relate to quantifiable phenorriena, whereas the 
others are compound indicators that measure qualitative phenomena by ranking 
their constituent parts in relation to predetermined standards. 

The indicator for the proportion of hours of teaching actually provided to a 
pupil makes it possible to assess, firstly, the number of hours lost owing to pupil 
absenteeism and, secondly, the number of hours lost owing to the non-replacement 
of teachers absent for reasons connected with the way the system operates, for train- 
ing, for personal reasons or because the school is closed. 

The indicator that measures the level of services related to life in the school 
makes it possible to assess the resources available to the school in respect of; 

• hours of attendance of educational advisers, medical and paramedical staff, 
and school social workers; floor area of heated rooms open to pupils during 
the midday break; floor area of the boarding facilities; seating capacity of the 
school canteen; 

• the significance of other activities as measured by the number of pupils regis- 
tered in one or more clubs and number of members of the sports association. 

Each of these elements is expressed in relation to the total number of pupils at the 
school and compared with benchmark standards or values provided by the ministry. 

The indicator that measures the level of teaching-related services makes it pos- 
sible to assess the services offered by career officers, psychologists whose function 
is to help pupils identify their vocational choices and formulate them more clearly. 
It also expresses the seating capacity of the documentation and information centre 
in relation to the total number of pupils, as well as the number of hours it is open 
with a documentalist in attendance. The latter element reveals the importance attached 
by the school to this service, which provides better conditions of study and work 
for pupils. 

Indicators related to the local context 

Two indicators are provided: one gives a general idea of the socio-economic fabric 
of the economic area in which the school is located (unemployment rate, descrip- 
tion of active population, overview of means of production, population and housing 
data and data on pre-service training); the other one measures the school’s level of 
integration into the local economy and brings out, through the nature and frequency 
of exchanges with local companies, the school’s dynamism and its involvement in 
the integration of its pupils into the economic environment. 
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Indicators of results 

The results obtained by a school are considered from different points of view: the aim 
is not to focus solely on success in examinations but to cover the entire school career 
of a pupil. Thus, gross pass rates are given by branch of study and by age, together 
with rates of access to higher education and diplomas, while taking account of redi- 
rection during secondary education towards other schools or into working life. 

At the same time, these indicators aim to apply an ‘added value’ approach by 
comparing gross pass rates with ‘expected’ rates, that is to say, the rates that would 
be achieved if the pupils entered for the examination had succeeded in the same pro- 
portion as all comparable pupils in the academic or in the country as a whole. 

Also included are indicators designed to evaluate the guidance provided by 
noting what subsequently happens to pupils who have dropped out of a particular 
level or left the school. 

PREPARATION AND DISSEMINATION OF THE IPES SYSTEM 

Three types of indicators can be distinguished by how and where they are drawn 
up: ‘analytical’ indicators calculated at the ministry; ‘progress-report’ indicators 
(indicateurs de ‘ compte-rendu’) and ‘individualized’ indicators worked out by the 
school concerned. 

All of them are primarily intended to provide the school head with informa- 
tion helpful for management purposes, by means of the IPES software which is sent 
each year to each school at the time it is issued. 

Those in positions of responsibility in the education system — officials of the 
academic^ school inspectors and ministry officials — are given access to the 
‘analytical’ indicators alone. They can be consulted on the Intranet, using a pass- 
word provided by the computer system managers. 

Only three indicators are made public. These are the performance indicators 
of general and technical lycees and vocational lycees^ namely, the baccalaureat access 
rate, the baccalaureat pass rate and the proportion of successful baccalaureat can- 
didates among school-leavers. These rates are accompanied for reference purposes 
by national figures and figures for each academic^ and by ‘expected’ rates, which 
make it possible to measure added value; they may be consulted on the Internet site 
of the Ministry of National Education. 

The 'analytical' indicators 

These are calculated by the ministry from the computer files used for the manage- 
ment of pupils, for the management of teachers and for school structures. In each 
college (lower secondary school) and public lycee, the same software (GEP — ges^ 
tion des eleves et des personnels — management of pupils and staff) has been 
introduced, whereby files work their way regularly up towards the national author- 
ity via the regional academie. The aggregation of data from these computerized 
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sources makes it possible to prepare regional and national statistics, which are in 
return made available to the schools, thus enabling them to see where they stand in 
a broader geographical context. These indicators, which are common to all schools 
of the same type — collegesy lycees and vocational lycees — are chiefly intended to 
assist overall management. They are accompanied by reference data for the nation 
as a whole and for each academie^ enabling the school to place itself in the wider 
context. They concern data for the current year, but also include time series, thus 
facilitating a historical approach. 

Schools can access these indicators by means of the consultation function of 
the IPES software supplied to them; they are also available on the ministry’s Intranet, 
where they may be consulted — with the necessary password — by authorized staff at 
the various levels of national education {departement, academic^ central adminis- 
tration). They cover the student population (sex, profession and social category of 
head of family, age, educational retardation, grade repeating, previous schooling, 
nationality, grants, etc.), resources (information on teachers, courses offered, lan- 
guages taught, structures, etc.), general organization and environment (size of classes, 
stability of teaching staff, data concerning the local employment area, etc.), and 
results (pass rate in examinations, access to higher education and diplomas, subse- 
quent career of pupils with or without qualifications), 

*Progress-reporf indicators 

These are calculated by the software package at the school itself, on the basis of data 
collected locally and not available in the national files. They relate to certain orga- 
nizational aspects of the school, and are generally for internal use, but may be 
transmitted — though this is not compulsory — to the authorities of the academic. 
These indicators were designed in particular for reporting to the school’s governing 
board on the running of the school and its results during the previous year. They 
are accompanied by national benchmark figures based on data derived from an 
annual survey of a national sample of schools. 

They are focused essentially on the set of qualitative indicators dealing, for 
example, with the closeness of relations with the economic environment, the level 
of ancillary or teaching-related services or the status of pupils. They are also used 
to measure some quantitative indicators such as school accident rate, popularity of 
the school, proportion of timetable hours when pupils are not being taught, etc. The 
analysis of all these indicators enables the climate of the school to be assessed. 

The indicator of ‘hours without being taught’ can be used to illustrate this type 
of indicator. It gauges one aspect of how the school is functioning, namely the actu- 
al amount of time during which an average pupil is being taught in relation to the 
theoretical amount of time planned at the beginning of the school year. In other 
words, it measures the amount of time lost by the pupil, not only because of his own 
absences, but also because of the non-replacement of absent teachers or closure of 
the school. The time ‘losses’ in fact need to be dealt with differently according to 
their different causes, of which the indicator distinguishes four: 
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(a) losses due to the way the system operates: organization and holding of exam- 
inations or student counselling sessions that might or might not entail the 
closing of the entire school, delays in the appointment of teachers, or closing 
of the school by decision of the authority or owing to circumstances outside 
its control, etc.; 

(b) losses due to the in-service training of teachers — essentially non-replaced absences 
of teachers attending training courses; 

(c) losses due to the non-replacement of teachers absent for personal reasons (ill- 
ness, maternity, strikes, etc.); 

(d) losses due to absence of the pupils themselves. 

'Individualized' indicators 

The software supplied with the indicators provides facilities for creating individu- 
alized variables and indicators geared to the specific monitoring operations that a 
school might wish to carry out. These indicators, specific to each school, are designed 
by the school head and calculated by the IPES software. 

They may, for example, cover the use made of the documentation and infor- 
mation centre or disciplinary measures, with figures concerning the number of pupils 
sent before the disciplinary board, temporarily excluded or expelled. 

Assessing the 'performance' of lycees: 

the special role of the 'analytical' indicators 

The best-known in this set of indicators are the ones relating to results, the reason 
being that they can be accessed by the general public. Parents may. consult them 
when looking for the most suitable school for the particular needs of their child, 
though their choice will be limited by school zoning rules based on the catchment 
area in which they live. They also provide material for discussions within the school 
and with the school’s partners. 

They are based on examination results, individual school careers, the propor- 
tion of school-leavers with diplomas, and what becomes of pupils after they have 
left the school. They make it possible to measure the school’s ‘added value’ because 
they allow the results obtained to be compared with expected results. In the tables 
below we take the case of a lycee^ a school whose essential purpose is to prepare its 
pupils for the baccalaureate to illustrate these indicators. 

Thus, the assessment of the performance of a lycee on the basis of its results 
brings several indicators into play — baccalaureat pass rate, baccalaureat access rate 
from the first year of upper secondary education, baccalaureat access rate from the 
second year and proportion of those leaving the lycee with a baccalaureat — the first 
three of which make it possible to assess the school’s added value in comparison 
with the figures for the academie and for France as a whole. They make no claim 
to measure the impact of the school as such but make for a more just appreciation 
of its results. 
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Table 1. List of indicators available to a general and technical lycee 



Structure of the school: indicators of student population 
pi — overall population at start of school year; 
p2 — pupils present in lycee classes at start of school year; 
p3 — previous schooling of pupils attending the lycee, by level; 
pS — level of knov^ledge of first-year pupils. 

Indicators of resources and means 
ml — number of timetable hours; 
m2 — characteristics of teachers; 
m3 — number of pupils in each course. 

Indicators of results 

rl — baccalaureat pass rate; 

rl — baccalaureat access rate; 

r3 — proportion of successful baccalaureat candidates among school-leavers; 
t4 — destination of pupils leaving the lycee^^ 
r6 — situation of school-leavers. 

Indicators of organizational aspects 

fl — direction taken by pupils at end of first year; 

fl — proportion of timetable hours when pupils are actually being taught; 
f3 — size of divisions; 

f4 — provision of teaching-related services; 
fS — provision of ancillary services; 
f6 — changes in composition of teaching staff; 
f7 — accidents in the school; 
f8 — status of lycee students. 

Indicators related to the local context 

el — closeness of relations with economic environment; 

el — data per employment area. 



BRINGING OUT THE SCHOOLS’ ‘ADDED VALUE’ 



To gauge the real contribution of the lycee to the success of its pupils, that is to say 
its added value, the influence of external factors needs to be eliminated and only 
that resulting from the school’s own action has to be taken into account. 

Several factors directly related to the pupil have an impact on academic suc- 
cess. Preliminary studies carried out at the Ministry of Education have emphasized 
the importance of two factors: the age of pupils and their social background. As the 
existence of strong correlations between those factors and academic success has been 
proved, the problem was to measure success in the school while neutralizing the 
effects of those two criteria. 

Apart from their relevance, these criteria were chosen because of their avail- 
ability in the information systems. The age criterion, for example, comes in handy 
as a substitute for an academic criterion that is difficult to measure, namely the level 
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of knowledge of pupils starting the first year of lycee. It is also well known that edu- 
cational retardation is an expression of learning difficulties. As for socio-professional 
category, the findings of several studies have shown the importance of this criteri- 
on and its impact on success throughout the school career. 

Thus, two expected rates are calculated, one based on all schoolchildren in the 
academie in which the school is situated (the academie-hzst expected rate’) and the 
other based on all schoolchildren in France (the ‘national-base expected rate’). The 
comparison of these expected rates with actual rates makes it possible to assess the 
added value and to determine the actual contribution of the school to the success 
of its pupils in comparison with the success of pupils in similar schools with student 
populations comparable with those of the academie and France as a whole. It is a 
relative assessment in that it compares the actual success — whether baccalaureat 
passes or access to the baccalaureat of first-year pupils — with an imaginary expect- 
ed success that the school would have obtained if its pupils had had the same results 
as those observed in an ‘average’ school in the academie and in France as a whole 
with a similar population and offering the same types of training. 

For example, the national average for the baccalaureat pass rate in 1999 was 
78.4%, but if we take the national average pass rate of candidates of normal age or 
younger from a very favoured social category, the rate rises to 90.1%. On the other 
hand, when we look at candidates two or more years behind in their studies and 
coming from a disadvantaged social category, the average pass rate is 65.2%. Under 
these conditions, a lycee whose average pass rate is 70% and whose candidates are 
almost all two or more years behind and from disadvantaged social categories could 
take legitimate satisfaction in the result obtained since, at the national level, the pass 
rate for students of the same age and social category was only 65.2%. Conversely, 
a school entering only students of normal age from a very favoured social catego- 
ry, could not be happy with a pass rate of 80% since that rate, although higher than 
the national average (78.4%), is markedly lower than the national pass rate for stu- 
dents with the same characteristics (90.1%). 

It would therefore seem more to the point for a lycee to compare its pass rate 
not with the national average rate but with an expected rate that takes account of 
the characteristics of its pupils by age and social category; each resulting group 
would then be compared with the national average pass rate for pupils with the 
same characteristics (see Table 1). The difference between the two figures serves to 
measure the added value of the school concerned. If the difference is positive, the 
lycee can reasonably be regarded as having given its pupils more than they would 
have received if they had attended an — imaginary — school reflecting the national 
average or the average of the academies this would indicate good performance, while 
a negative difference would indicate the opposite. 

To illustrate these performance indicators, a school has been chosen at ran- 
dom and its results presented in Tables 2, 3 and 4. 
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Table 2. Baccalaureat pass rate, by specialization, of a given school 



Specialization 


Literary 


Economic 
and social 


Scientific 


Tertiary 
science and 
technology 


Industrial 
science and 
technology 


Laboratory 
science and 
technology 


Overall 

rate 


Gross rate 


87 


- 


85 


- 


82 


82 


84 


Expected rate 
(for academie) 


. 88 


- 


83 


- 


83 


78 


82 


Expected rate 
(for France) 


83 


- 


79 


- 


78 


80 


79 


Total number 
of candidates 


39 


- 


102 


- 


96 


71 


308 



Note: This table shows that the school concerned obtains an overall baccalaureat pass rate 
of 84%, slightly higher than the expected rates for the academie (82%) or for France (79%); 
its added value is thus positive. However, analysis of the performance of this school should 
be supplemented by measurements of its access rates in order to determine the true success 
rate for a full school career (see Table 4). 



EVALUATION BASED ON SEVERAL PARAMETERS 

If, as mentioned earlier, we seek to measure the ‘added value’ of a school, it emerges 
that success in the school-leaving examination, even when compared with an expect- 
ed rate rather than an unadjusted average rate, cannot be used as a school’s sole 
performance indicator: attention should also be paid to the pupils’ whole career in 
the school and to the proportion leaving the lycee with their school-leaving diplo- 
ma, even if they have had to repeat a grade or grades. The latter rate, ideally 100%, 
makes it possible among other things to assess the school’s ability to offer a ‘second 
chance’ thanks to grade repeating that leads to eventual success. We know that this 
is not always the case, and that, each year, a number of pupils leave school for good 
without the longed-for diploma (Table 3). 

Table 3. Proportion of successful baccalaureat czndidztts among school-leavers of a given lycee 





School % 


Academie % 


France % 


First, second and third years 


58 


75 


69 


Third year 


95 


94 


93 



Note: This table shows that the proportion of baccalaureat-holders among those leaving 
school at the end of the third year is satisfactory, but is not satisfactory for school-leavers 
from the first, second and third years taken together. 

ZQ 
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Let us now look more closely at the educational career of a pupil in a given 
lycee^ from entering the school or a given branch of study until he or she leaves hav- 
ing obtained the certificate or diploma originally aimed for, however many years 
were needed to obtain it. The chances of success, during the pupils’ career spent in 
the same school, are expressed by ‘access rates’ from the first year of lycee to the 
baccalaureat or from the second year (i.e. second year of lycee but year of entry into 
the chosen branch of study) to the baccalaureat (Table 4). 

The access rate from the first year of lycee to the baccalaureat is calculated on 
the basis of the access rate from the first to the second year, the access rate from the 
second to the third (and final) year, and the access rate from the third year of lycee 
to the baccalaureat. 

It is arrived at by following an ‘imaginary cohort’, the calculation being based 
on lycee pupils at all levels in a given year, and on the assumption that the obser- 
vations for a given year, in terms of access from one level to the next and final success 
in the baccalaureat examination, can be extrapolated to the entire career of a pupil 
in the school. 

For these access rates, as for the baccalaureat pass rate, consideration is also 
given to the expected rates calculated with the characteristics of the student popu- 
lation taken into account. The comparison of those expected rates with actual access 
rates makes it possible to measure the added value. 

Table 4. Access rates from the first and second years to the baccalaureat for a given lycee 





Gross rate 
(academie) 


Expected rate 
(France) 


Expected rate 




Total number 
of pupils 


First year 


50 


69 


62 


First year 


374 


Second year 


80 


83 


81 


Second year 


338 



Note: This table shows an access rate from first year to baccalaureat much lower than the 
expected rates, but not from the second year. 



At all events, it is the cross-referencing of all these indicators of outcomes — 
success in the baccalaureat^ access rates and proportion of baccalaureat-holders 
among school-leavers — that gives the best idea of a school’s success in relation to 
the objective of obtaining a school-leaving certificate. Noting the baccalaureat pass 
rate alone, or even compared to an expected rate, would mean failing to take into 
account the policies of certain schools which turn away pupils they consider unlike- 
ly to be able to obtain the diploma aimed for; this could produce very satisfactory 
pass rates, but at the cost of a policy of selection at an earlier stage that gives little 
chance to pupils in temporary difficulty or needing more time to attain the requisite 
level. 

Thus, in the case of the lycee whose results are set out in the previous three 
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tables, the indicators taken together might suggest a selective policy at the end of 
the first year, with the elimination of the weakest pupils, resulting — at this price- 
in satisfactory results in the final examination. Another hypothesis is, however, 
possible: as can be seen in the first table, the school does not offer ‘economic and 
social’ or ‘tertiary science and technology’ courses of study. That might have led a 
number of pupils interested in those courses of study to leave the school for quite 
legitimate reasons at the end of the first year, when the choice has to be made. The 
low access rate from the first year to the baccalaureat would then be more an effect 
of ‘structure’ than the result of a policy of selection. At all events, the only way of 
confirming either of these hypotheses or even checking a third one would be to obtain 
additional local information. 

A system that is clearly useful 
but not sufficient in itself 

The battery of indicators presented here is a crucial component in the development 
of procedures to assist heads in the running of French secondary schools. The pro- 
vision of pre-calculated data, along with national and regional cross-references and 
a software package giving access to these data, is greatly appreciated by school heads, 
but however essential this system may be, it is not sufficient in itself, since other ele- 
ments are equally necessary: 

® The development of a minimal level of computer skills on the part of school 
heads is vital for the dissemination and rational use of this set of indicators. 

® The development of an evaluation culture: the use, interpretation or mere 
perusal of indicators requires the acquisition of certain basic concepts and 
methods that will avert errors of interpretation. The skills of critical appraisal, 
comparison, crosschecking and postulating of hypotheses may be developed 
by means of pre-service or in-service training aimed at those responsible for 
running schools. This is what has been done systematically since the intro- 
duction of IPES in 1994, but it should also include ‘booster’ courses for school 
heads in need of them. 

® Outside assistance and an outside view: this refers to external evaluation, which 
can provide a very useful complement to the school’s internal evaluation. This 
is the job of the inspectorate. Inspectors can use the IPES indicators to formu- 
late a number of hypotheses which, in any case, will need to be checked by a 
visit to the school and discussions with its head and the teaching team to cast 
more light on its operation and the choices made by it. 

The experience gained over a period of seven years has highlighted the great impor- 
tance of the school head’s personal involvement: this involvement must clearly be 
intense, and his or her personal work must be done with conviction if the delicate 
task of steering schools with the help of these indicators is to be successful. That is 
why the following two conditions seem of decisive importance: 

® The determination to steer a project-based course, primarily on the part of the 

28 

Prospects, voi XXXI, no. 4, December 2001 




IPES in France 



487 



school head, whose task it is to convince the educational community, espe- 
cially the teachers, of the effectiveness of pursuing such a course, 
o Leadership: the school head has a vital role to play in promoting the use by 
the school of indicators to facilitate evaluation and management. The head’s 
position gives him or her control over the information, the means of dissemi- 
nating it and the initiative in using it as he or she wishes. It is up to the head 
to present it to the members of the educational community, especially the teach- 
ers, to analyse it together with them, to steer them towards the definition of 
assessable objectives, and to assist in devising and subsequently putting into 
practice educational projects and teaching strategies conducive to achievement 
of the objectives set. Lastly, it is the head’s job to conduct the evaluation that 
will enable the school to fine-tune, or make radical changes to, its policy. 

In conclusion, it is worth mentioning that, in keeping with the thinking behind the 
introduction of this system of indicators, an information centre has recently been 
set up by the French Ministry of Education. Access is via the Ministry’s Intranet net- 
work. It includes various sources of data, such as document and decision databases, 
indicators and directories produced by the central administration. These data are 
used by all education system administrators. The available tools for evaluation and 
management include, in addition to the IPES, a ‘central management base’ [base 
central de pilotage — BCP), ‘management tools for regional and academie-level edu- 
cational institutions’ (outils de pilotage des enseignements regionaux et 
academiques — OPERA), ‘indicators on higher education institutions’ (indicateurs 
sur les etablissements d* enseignement superieur — InfoSUP), and ‘indicators for steer- 
age from primary to lower secondary school’ [indicateurs pour le pilotage de Vecole 
au college — InPEC) the object of which is to improve transition from primary to 
lower secondary education. 

Thus, the whole of the French education system is provided with indicators 
that supply all the senior staff of its various institutions with objective information 
to help them carry out their duties in regard to evaluation and management as well 
as possible. 

Notes 

1. For the administration of its education system, France is divided into thirty geograph- 
ical regions known as academies^ each headed by a rector. 

2. Baccalaureat pass rates and the school ‘league tables’ based on them were published 
annually in Le Monde de V education until 1994. However, when the lycee performance 
indicators began to be issued by the Ministry, the press ceased to rank schools solely 
on the basis of pass rates for the baccalaureat. 

3. The lycee performance indicators can be found on the Internet site http://www. educa- 
tion. gov.fr/ival . This site is updated in April each year. 

4. The first of four years in lower secondary education. 

5. The first of three years. 

6. The first of two years. 
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'VALUE-ADDED' MEASURES 
IN SCHOOL EVALUATION: 
A VIEW FROM ENGLAND 

Lesley Saunders 



This paper is about the type of measurement of schools’ performance that has come 
to be known as ‘value added’. In the first main section, the paper gives an account 
of the context for value-added approaches in education, and provides some criteria 
for calculating and presenting value-added measures appropriately. The second half 
of the paper discusses the implications of a study undertaken to explore how schools 
actually use value-added information and concludes by suggesting that ‘the psy- 
chology and sociology of using numbers’ need to be better understood. 

An overview of 'value-added' approaches 

‘Value-added’ measurement of educational performance is here to stay or else on 
the way in the education systems of several countries. In the United Kingdom, and 
more particularly England, for example, the national value-added system — ^instituted 
in 1998 — at the moment produces information for primary and secondary schools 
‘to assist schools in looking at their own pupils’ progress against the national pic- 
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ture’, based on national curriculum test and public examination results. As soon as 
it is feasible, the government proposes to include value-added measures in the tables 
that publish information about the performance of every school in the country. As 
I argued in a recent report (Saunders, 1999b), the introduction of the national value- 
added system could be seen as the culmination of a decade of sustained and fairly 
public argument about how to measure the performance of pupils in the nation’s 
schools in a way that sheds light on progress as well as standards. 

Measured at the school level, value added is the most accurate way we have 
at present of calculating how well schools perform with their pupils. If only ‘raw’ 
results are used to assess a school’s performance, they reveal more about the back- 
ground of the pupils than about the performance of the school. Value-added 
measurement works by discounting factors tied into pupils’ achievements but unre- 
lated to institutional quality. Such factors include pupils’ prior attainment, sex, 
ethnic group, date of birth, level of special educational need and social disadvan- 
tage. These factors — whether correlated positively with performance (as in the case 
of prior attainment) or negatively (as in the case of social disadvantage) — turn up 
as empirically verifiable items in study after study. 

However, this sharpened focus on more sophisticated ways of evaluating schools’ 
performance — which may have far-reaching consequences for the education system, 
the schools and their pupils — means that it is crucial to get the calculation, inter- 
pretation and uses of ‘value-added’ data right. 

What is the context for 'value added' 

The context and rationale for calculating value-added measures of performance have 
changed dramatically since the 1980s, when ‘value added’ was regarded as a quasi- 
technical idea that had strayed into education from economics. Educational ‘value 
added’ is one of those terms that comes with an agenda already attached: the agen- 
da in this case being the political preoccupation in the United Kingdom (as in other 
nations) with standards and quality in education, or rather the lack of quality and/or 
of deterioration in standards therein and the wish of politicians to get better value 
for public expenditure. For some time, this agenda has been explicitly attached to 
the issue of global competitiveness, and the consensus belief that education and 
training are important levers for economic competitive advantage. For their part, 
education managers and school principals have often wanted to demonstrate that 
standards (particularly in their own institutions) have not been slipping, that the 
profession is giving value for money. 

One could say that value added was an idea waiting for its time: the ideas and 
methodological models that lay behind the introduction of value added in educa- 
tion were already well established by the early 1990s. But when value-added 
approaches began to be talked about outside the circle of educational scholars and 
statisticians, the then secretary of state for education in England was scathing about 
what he was convinced were ever more sophisticated ways of ‘cooking’ schools’ 
results: he wanted to stick with ‘raw’ ones that were simple to compile and under- 
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stand. By 1995, political opinion had changed dramatically: the next secretary of 
state for education (and employment) had been won over to the principle of value 
added to the extent that she established the framework for a national system. That 
same year, a research contract was awarded to the Centre for Educational 
Management (then based at the University of Newcastle), by what was the Schools 
Curriculum and Assessment Authority (SCAA), to design and pilot methods of cal- 
culating value added and ways of using the results to assess schools’ performance. 
(See Saunders, 1999^, for a fuller discussion of these issues.) 

Various methods have been — and continue to be — tested and developed for 
ways of producing information that can be used to assess the relative effectiveness 
of different schools or local education authorities (LEAs). See, over the years, the 
work accomplished by Fitz-Gibbon, Goldstein, Gray, Hutchison, Kendall, Jesson, 
Mortimore, Nuttall, Sammons, Schagen, Thomas, Tymms and others: for example, 
in Gray et aL, 1986; Raudenbusch and Bryk, 1986; Nuttall et aL, 1989; Gray et aL, 
1990; Tymms, 1990; Schagen, 1991; Fitz-Gibbon and Tymms, 1993; Hutchison, 
1993; Mortimore et aL, 1994; Gray et al., 1995; Kendall, 1995; Jesson, 1996; Thomas 
& Mortimore, 1996; Sammons et aL^ 1997. 

How can these measures improve the system? 

It is important to realise that collecting and analysing data that can be used for 
‘value-added’ purposes is costly in terms of resource and expertise. (See Goldstein, 
1997, for discussion of how far we still have to go in making best use of the tech- 
niques.) Many items of standardized performance and background data must be 
collected, linked to a unique pupil ID and analysed using a statistical technique called 
multilevel modelling. 

The question of whether this is an investment worth making must be answered 
in terms of the uses to be made of such analyses, and of the quality of the inspec- 
toral/advisory infrastructures to support such usage. The potential uses of value 
added can be broadly grouped into ‘accountability’ and ‘improvement/development’. 
Although in principle these are not mutually incompatible, in practice there is usu- 
ally a pronounced emphasis one way or the other in policy preference and resource 
allocation. 

ACCOUNTABILITY 

The main need in a system that emphasizes accountability — and this is often allied 
with support for the ‘marketization’ of schools through the unfettered exercise of 
parental choice — is for clear and simple indicators giving unambiguous information 
that differentiates between schools. A problem with using value-added analyses for 
accountability purposes is that they typically show that — on any given indicator — 
most schools are performing within the range that might be expected given their 
intake characteristics. Value-added analyses, therefore, cannot sensibly be used to 
‘rank order’ all schools. Indeed, it is hard to make a case for using value-added mea- 
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sures — or any other performance indicators — to arrive at simple definitive state- 
ments about schools’ effectiveness. 

Value-added analyses can be used, however, to identify the ‘outliers’ or excep- 
tions — i.e. those schools which appear to be performing significantly better or 
significantly worse than expected. Once identified, these institutions can — assum- 
ing the appropriate inspectoral/advisory infrastructures are in place — be further 
investigated in relation to the quality of teaching methods, climate, leadership, etc. 
This information could yield insights into the operation of the system as a whole. 

IMPROVEMENT/DEVELOPMENT 

But perhaps a more important issue is how far are the complexities that have been 
introduced to the measurement of educational quality by value added justified in 
what they have contributed to the possibility of raising standards in individual 
schools,^ The main need in a system that emphasizes improvement/development is 
for quite detailed analyses that can compare and contrast, within one school or 
municipality, the performance over time of pupils in different subject areas, for 
example, or of different groups of pupils in the same subject. Schools can use these 
analyses as part of their self-evaluation process to diagnose institutional strengths 
and weaknesses. In this context, value-added measures can help to pose better and 
more focused questions about the ways a school is performing with its pupils and 
to stimulate more informed discussion amongst school staff about the way they 
organize and deliver their teaching. 

But we can hypothesize that to perform this function well, value-added analy- 
ses need to be treated as an educational innovation: i.e. actively managed with 
adequate preparation, support and training for school governors, principals and 
teachers. Some commentators would add that the analyses must be confidential — 
i.e. not released into the public domain — in order to realise their potential for 
developmental objectives. 

What are the basic technical features? 

A brief summary of points to bear in mind might go as follows:^ 

1. Value-added indicators are an improvement on raw results, to the extent that 
the latter say a lot more about the intake than about the effectiveness of a 
school. Value-added analyses attempt to strip away factors that are associat- 
ed with performance — either positively or negatively — but are not related to 
institutional quality. These include pupils’ prior attainment, sex, ethnic group, 
date of birth, level of special educational need and social disadvantage. It is 
not unusual for pupils’ prior attainment plus the overall level of social disad- 
vantage in the school to account for as much as 80% of the apparent difference 
between schools. 

2. Any quantitative data — whether raw or value added — has the following limi- 
tations: 
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® Quantitative data carries with it a degree of statistical ‘uncertainty’: calcula- 
tions based on numerical data can be as much the result of pure chance as of 
something more ‘real’, such as the quality of education. Usually, only a few 
schools can be shown to be performing at levels significantly above or below 
the norm in statistical terms, on either raw or ‘value-added’ outcomes. 

• Analyses of performance data — because necessarily retrospective — can easily 
be misleading with regard to current and future performance. 

• Academic performance — which is the usual outcome measure used in value- 
added calculations — is not the only (nor even, according to some educators, 
the most important) outcome of ten or twelve years of schooling. 

• Schools may be ‘differentially effective’: 

First of all, different outcome measures often give different results: a school 
that is getting good results if one looks only at higher levels of attainment may 
be letting down those pupils at the lower end of the attainment range; 

Or, a school may be doing well with some groups of pupils (girls in some eth- 
nic groups, for example) and not others (boys in other ethnic groups); and/or 
in mathematics but not English. 

• The national statistical relationship between, say, test results at the end of pri- 
mary education and test results at the end of secondary education may change. 
To maintain their accuracy and relevance, statistical models must be continu- 
ally monitored by empirical research. 

3. Most of this has been obscured by crude ‘league table’ approaches, however. 
One of the strengths of value-added measures has been to make educators more 
aware of the complex and probabilistic nature of performance data. 

4. The prerequisites for doing ‘value added’ well are illustrated in Box 1. 

VALUE ADDED IN BRIEF 

Value-added analyses: 

• Are only as good as the data they are based on; 

• Deal in correlations, not causes; 

BOX 1. Best practice in calculating value added. 

• Data collected at the pupil level on a large and representative sample. 

• Outcome measure(s) reflecting all levels of pupil performance. 

• Prior attainment measure(s) for each pupil (preferably individual standardized 
scores), plus items of information about the pupil’s background. 

• School context factors. 

• Multilevel modelling to analyse the data. 

• Rigorous quality assurance and quality control procedures. 

Source: See: Gray, 1 994. 
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BOX 2: Best practice in providing value added, 

<5 Results of analyses should be given in tabular, graphical and verbal forms for 
individual schools. These results should show whether the measured differences 
between schools are statistically significant, 
o Wherever possible, results should be given at three stages of analysis: ‘raw’ 
results, results adjusted for pupil data (such as prior attainment) and results 
adjusted for pupil and school context data. Each one of these stages of analysis 
usually has some insight to offer. 

o A minimum of three years’ worth of data is needed to establish a trend upward 
or downward. 

o It follows from the above that value-added measurements are not self-evident 
nor without their limitations. A detailed explanation needs to accompany the 
analyses. 

® Are based on a normative and retrospective model that may not tell us any 
thing about desirable levels of performance now or in the future; 

® Are only one instrument of evaluation; 

® Should be used as a screening device, not a definitive assessment; 
o Rigorous quality assurance and quality control procedures. 

® Provide no right answers or quick fixes to school improvement; 

® Are therefore necessary but not sufficient for understanding how well schools 
are performing; 

An important corollary of all this is that the ‘high stakes’ use of value-added analy- 
ses is not supportable, any more than is that of crude league tables. Value-added 
measures would not make what is essentially an inappropriate use of performance 
data more acceptable just because they use better methods of calculation. By them- 
selves, they cannot be validly used to make definitive judgements about the 
effectiveness of a school, still less of a single department or class or teacher. (See the 
article by Visscher & Karsten in ProspectSy vol XXXI, no. 2 for an excellent dis- 
cussion of the uses of school performance indicators; and Goldstein &c Myers, 1996, 
for a well-argued rationale for a code of ethics governing the use of performance 
indicators.) 

The remainder of this paper discusses the findings of a small study that explored 
the uses of value-added information by schools. These findings serve both to illus- 
trate some of the issues raised above and point to some challenges to our 
understanding of what happens to ‘evidence’ in an institutional context. 

Schools^ use of value-added data 

WHAT IMPLICATIONS FOR POLICY.^ 

If the national emphasis in valued added (and performance data generally) is now 
on supporting and improving the quality of teaching and learning, what kinds of 
evidence do we have on the ways in which school staff do actually use value-added 
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data and what does it show? What kinds of professional development and support 
are necessary? What capacity does value added have for raising pupil performance? 

For staff at the National Foundation for Educational Research (NFER), the 
guidance and in-service training aspect of the value-added work we undertake has 
been particularly instructive. Quite rightly, the focus of work in the school effec- 
tiveness and value-added fields has tended to be methodological accuracy — from 
conceptualizing the statistical model appropriately to collecting the right data in the 
right form. What use headteachers and staff made of the analyses was something 
to which we perhaps did not initially give full enough consideration. We were not 
alone in this. The guidance given by national agencies on the use of data for target- 
setting and raising standards has in the past been premised on assumptions about key 
factors, such as the transparency of data and the part played by the internal orga- 
nization and culture of schools. This second section has something to say about 
what might be called the ‘user end’ of the process. 

To begin with, we need to be aware that not all school staff may have been 
able to acquire an accurate understanding of value added. Maw (1999) argues that 
the press coverage of value added has largely been in the context of publishing the 
controversial tables of school performance, and has not given a clear and coherent 
message concerning the measurement of value added. 

There are certainly indications that the use of value-added data by school staff 
is not a straightforward matter of applying ‘evidence’ to practice. The NFER recent- 
ly funded a small-scale qualitative study to explore in a preliminary fashion, through 
interview, observation and documentary analysis, the uses to which QUASE'^ data 
was being put in nine secondary schools in different local education authorities. 

The first point to make about the study is that the circumstances were appar- 
ently highly conducive to schools making use of value-added information, in that: 

• The schools — or rather, the headteachers/governors — had commissioned and 
paid for the analyses (rather than having the information imposed on them); 

• The information was of direct relevance for the individual school (as distinct 
from information about national data for which they would have to generate 
their own comparisons); 

• The information was accessible in the sense that the report was sent to the 
school (as distinct from the school having to seek out information, for exam- 
ple, from the Internet); 

• The information was important for the (policy-driven) focus on performance. 
The study found that in some schools the data was being disseminated widely and 
used constructively. This was more likely to happen if staff had opportunities for 
constructing collective meanings through discussion and dialogue about the data. 
Nonetheless, despite the presumably favourable pre-conditions in these schools, a 
variety of challenges and difficulties seemed to be in the way of at least some staff 
in all the schools using value-added data to its fullest extent and for developmental 
purposes. Looking at the fieldwork evidence in one way, the possible reasons for 
this can be surmised to be: 

® Lack of structures for dissemination of the information; 
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o Lack of time for staff to read and interpret the information; 

® Lack of competence/confidence amongst staff to understand the information; 
o Lack of inclination by staff to spend time or acquire competence in using an 
instrument whose helpfulness to their work was not immediately obvious (Ht 
tells us nothing new'):, 

® Lack of belief in performance data as an aid to pedagogical excellence. 

These are the kinds of conclusions which a typical evaluation might draw and would 
suggest that value-added analyses have not (yet) played a large role as information 
for school improvement. But, plainly, this interpretation constitutes something of a 
deficit model. What else might the outcomes be telling us? What factors and forces 
might be at work? 

To begin with empirical evidence: The study found that only where all or most 
of the following conditions were present were schools likely to be making active use 
of value-added data. The five factors were: 

® Active support for value added by senior management; 

® ‘Championing’ of QUASE data by a senior member of staff; 

® The incentive provided by the school’s overall ‘relative effectiveness’; briefly, 
schools which had the greatest incentive to use value-added data were those 
doing poorly in league tables but well in terms of progress made with pupils; 
® At least three years’ participation in QUASE; 

® Previous exposure of at least some staff to guidance on the interpretation of 
QUASE. (These factors are discussed in more detail in Saunders & Rudd, 1999.) 
But I would go further and say that the evidence from the study — which, although 
small-scale, supports and extends the evidence from one or two other studies in the 
field (e.g. Williamson 8c Fitz-Gibbon, 1990; Williamson et al., 1992; Wikeley, 
1998) — suggests that there are a number of critical issues pertaining to the use of 
value-added performance data that deserve to be more widely understood. These 
can be expressed as follows: 

Value-added data carries a particular symbolism or ‘charge’ for many school 
staff at present. As many interviewees in the NFER study made clear, value-added 
data is seen as part of a rapidly moving policy agenda, about scrutinizing and mak- 
ing accountable school, departmental and teacher performance. Scrutiny of results 
for accountability purposes, at national, local education authority, institutional and 
sub-institutional levels, is likely to be strengthened, at least in the short term. 
Performance data generally may be impartial in intention, but can rarely be per- 
ceived as neutral in effect. Value-added data in particular is complex, relatively new 
and often has ‘high stakes’. It is probably not possible for any such data to be per- 
ceived and used in a way that wholly ignores — or remains unshaped by — this context. 

In any case, data do not speak for themselves. Data depend on a series of prior 
decisions about what has been measured and how it has been measured, decisions 
that necessarily involve value judgements as well as complex technical ones. These 
are not always transparent or understood. Partly because of this, some staff are 
unduly sceptical about performance data to the point of wanting to dismiss it; oth- 
ers are perhaps over-optimistic about what it can tell them. In the NFER study, 
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where value-added data was utilized well — i.e. as part of ‘intelligence-gathering’ 
about the school — it was often ‘championed’ by someone who understood and was 
interested in the data and was prepared to mediate the technical aspects for col- 
leagues; moreover, the data was used in conjunction with other information, both 
numerical and qualitative, to initiate discussion. 

Value-added analyses are evidently an intervention that needs to be managed. 
In the NFER study, senior managers in some schools were not sure how to make 
best use of the analyses, and let departmental heads ‘do their own thing’; this seemed 
to lead to inertia in some departments and ‘over-use’ of data in others. Other head 
teachers, especially those who had been using the data over several years, were more 
confident about what they wanted to achieve by the strategic use of data. Even so, 
there were still conspicuous differences in the use made by different departments in 
the same school. And so the question arises, is it not sometimes just as important 
to explore and understand why schools, or departments within schools, are not using 
value-added data? 

External input in the form of in-service training on value-added data seems to 
be a necessary prerequisite. In the NFER study, the input of previous training/sup- 
port was one factor associated with schools’ use of data. The study additionally 
suggested that the extent to which schools made use of the data was much in accor- 
dance with what might have been expected from their ‘relative effectiveness’, i.e. 
according to whether they had a strong extrinsic incentive to use it or not. From a 
policy perspective, this raises the question of whether further external encourage- 
ment and support from, for example, the local education authority may be needed. 
It also reinforces the idea that training and support that takes account of individ- 
ual school contexts is necessary in order for the potential of the data to be better 
understood and utilized. 

The meanings of data are socially constructed. The NFER study suggests that 
the meaning of value-added data emerges from the interaction among: 

® The actual ‘numbers on the page’; 

® The politicized significance which performance data as a species of informa- 
tion currently possesses; 

® What individual staff bring — often implicitly — to the discussion in terms of 
their own skills and knowledge about performance data, their values and atti- 
tudes towards schooling and teaching, their expectations and assessments of 
their pupils, their feelings about their place within the hierarchy of the school, 
and so forth. 

This last point in turn poses the question: Does value-added data sometimes act as 
a locus for conversations staff would like to have, or anxieties they want to express, 
about their professional work and its value in the organization? If so, then who con- 
trols (access to) value-added data and for what perceived purposes the data is being 
used are central rather than secondary questions. It has to be said that the use (or 
lack of use) to which value-added data are put seems to follow the existing contours 
of the style of management. It is not at all evident that the introduction of data nec- 
essarily and of itself brings about a culture supportive of systematic self-review and 
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planning for improvement; it seems rather the other way round, that the culture 
determines what use is made of the data. We might then want to ask, how well does 
the ‘champion’ or sponsor of value-added data manage and mediate the political 
and ethical as well as the technical aspects for colleagues? 

One English commentator (Dudley, 1999^, b) has some pertinent suggestions 
for helping teachers to deal with data. (In his case, he was discussing the use of pupil 
survey data rather than value-added data — but the general principles hold good.) 
He says that the factors that promoted an action-oriented response were found in 
the way the data were presented and the feedback managed. He notes that key issues 
within the data could easily be lost and that the discussion could be side-tracked or 
brought to a halt by staff focusing on variables over which the school had little con- 
trol, such as home background. 

Factors associated with positive, action-orientated responses were: 

• The availability of comparative data from schools felt not to be dissimilar; 

• Preparation for reading the data; 

• Prompts which focused discussion of pupil data on teaching, learning and issues 
that the school could influence and developed speculation skills in teachers; 

• Some prior groundwork done by the discussion leader in identifying possible 
improvement strategies to feed into the discussion so that people did not feel 
‘cornered’ by the data; 

• The introduction of ground rules into the discussion to ensure that apparent 
good news or reactions to perceived external influences such as home back- 
ground were sufficiently challenged; 

• A school where speculation and reflection were promoted among staff; 

• A feeling that the process generating the data was valid and could be trusted. 
(Amalgamation of Dudley, \999a, b.) 

In view of all this, we may well feel that ‘the psychology and sociology’ of using 
data need to be better understood and managed. The force of Dudley’s (1999^, 
p. 97) claim that ‘taking account of the emotional issues in effective use of data is 
vital if it is to be used with sustained effect' seems a crucial lesson to be drawn from 
the evidence. As well as better preparation for and management of value-added data, 
I think it is legitimate to argue that a better understanding is needed of what kinds 
of knowledge teachers feel moved and motivated by, and of which they can even- 
tually feel ownership. 

This is particularly important because a great deal is being said currently about 
the importance of ‘relevant’ research/data/information/evidence for both policy and 
practice (see, for example, Hargreaves, 1996). I said above that data do not speak 
for themselves: it is not self-evident. I think it is not going too far to say that the 
outcomes of the NFER study raise questions about the self-evidence of ‘relevance’. 
Bird et al. (1998, p. 35) define relevance in educational research as consisting of 
‘importance’ and ‘contribution to existing knowledge’. If we accept this definition, 
the NFER study suggests that, so far as schools and teachers are concerned, we 
should then ask ‘important for what and for whom?’ and ‘knowledge of what and 
for whom?’ 
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In other words, it cannot be assumed that the policy context of raising stan- 
dards — in which value-added data as a form of information should be 
situated — constitutes a necessary and sufficient definition of importance as far as 
teachers are concerned: indeed, they may feel dominated and demotivated by such 
policy initiatives. What teachers may then evince is ‘compliance with’ rather than 
‘ownership of. Nor can it be assumed that value-added analyses of performance 
constitute the kind of knowledge which is felt by teachers to contribute something 
worthwhile to what they already know or believe about the connection between 
pupils, progress and pedagogy. The convictions that some teachers and education- 
ists hold about this connection (and from which a scepticism towards value-added 
data could imaginably ensue) can be encapsulated in the words of Stenhouse (1975): 
'Education as induction into knowledge is successful to the extent that it makes the 
behavioural outcomes of the students unpredictable/ 

I accordingly wish to speculate that the NFER study points us in the direction 
of saying that why and how teachers engage with some kinds of knowledge and not 
others is bound up with: 

• Their assessment of the perceived costs in relation to potential benefits of engag- 
ing with complex and difficult subjects like value added, when there is already 
a large workload outside as well as within the classroom for teachers; 

• Their construction of professional identity, particularly as this concerns: 

The theories (albeit sometimes implicit) they have developed about their own 
learning as teachers; 

Their values- base: what they think it is most important for young people to be 
able to accomplish in their education; 

Their particular institutional culture and its micro-politics. 

Gray (2000, p. iv) even seems to insinuate that the notion of ‘relevance’ may itself 
be irrelevant to teachers’ learning: 'There are hardly any short-cuts — when it comes 
to learning how to change few teachers seem to be prepared to let others do the 
thinking for them \ 

These ideas suggest that the deficit model I presented above does not do jus- 
tice to the professional milieu of schools in terms either of teachers’ cognitive processes 
and their theories about those processes, or of the institutional and political con- 
texts in which they individually and collectively construct their cognition. And this 
in turn confirms the reasonableness of the frequently heard proposition that school 
improvement cannot be secured by ‘quick fixes’. Policy-makers may not be able to 
legislate for this, but they should understand, acknowledge and respect it. 

Notes 

1 . 

2 . 

3 . 



This section is based on an earlier article: L. Saunders and S, Thomas, Into an uncer- 
tain world. Times educational supplement (London), no. 4280, 10 July 1998, p. 26. 
A fuller account of the findings of the NFER study can be found in Saunders, 2000. 
QUASE — quantitative analysis for self-evaluation — is the value-added analysis service 
for secondary schools provided by the NFER. See Box 3. 
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Box 3: Work in value-added analyses at the National Foundation for Educational Research 

The National Foundation for Educational Research (NFER) has an established repu- 
tation in various aspects of educational research directly relevant to ‘value added’: 
notably, in educational test development, statistical analyses of performance, and 
assessment of school effectiveness. We have been undertaking ‘value-added’ work over 
the last decade in partnership with a number of schools and local education author- 
ities around the country. 

Our programme comprises two distinct but related types of work: evaluations 
and research studies sponsored by central government agencies or the NFER itself; 
and services to schools and local education authorities. 

Two large-scale evaluations of national initiatives are currently being under- 
taken for central government: 

0 The evaluation of the National Literacy and Numeracy Projects (autumn 1996 
to summer 2000); 

0 The evaluation of the National Literacy and Numeracy Strategies (spring 1999 
to autumn 2001). 

Both evaluations involve measuring progress made by pupils over time from a ‘base- 
line’ measure at the beginning of the initiative. Progress measures are calculated in 
terms of changes in standardized scores measured using specially designed tests of lit- 
eracy and numeracy. In each case, contextual data has been collected at pupil and 
school level to support sophisticated statistical analyses so that valid comparisons of 
the progress made by different groups of pupils may be made. Detailed feedback to 
schools on their pupils’ test results and progress scores is a key component. 

Another example of such work is the collection and analysis, for a large nation- 
al sample of pupils in 1998, 1999 and 2000, of the results of optional tests taken by 
pupils in years three, four and five in primary schools. The aim of this work is to pro- 
vide the Qualifications and Curriculum Authority (QCA) with national measures of 
performance on these tests and to identify and investigate any variation in perfor- 
mance by different groups of pupils. Participating schools have been provided with 
detailed reports of their pupils’ performance each year, including: 
o Levels achieved on the optional tests in relation to national norms; 
o Comparative performance of different groups of pupils on the optional tests, 
e.g. of boys compared with girls; 

® Progress made by pupils since the statutory national curriculum assessments at 
the end of key stage one in comparison with the pattern of progress in the nation- 
al sample. 

The response to these reports has been very favourable and schools appear to find the 
information helpful for tasks related to monitoring, planning and target-setting. 

Following consultation with local education authorities and the QCA, the NFER 
is now intending to provide an analysis service, on the lines described above, to inter- 
ested schools on a cost recovery basis. A question-by-question analysis of their pupils’ 
responses to the mathematics optional tests can be purchased by schools as part of 
the service: this generates a report linking test results to the various strands of the 
National Numeracy Strategy, and is intended as an aid to curriculum planning for the 
forthcoming academic year. 
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Other commissioned work in the general area of ‘value-added’ measurement 
includes the development of baseline assessment for pupils entering primary school. 

The NFER has also for some time provided services which provide value-added 
analyses for secondary schools which measure their performance in the light of key 
contextual factors, such as the prior attainment and socio-economic background of 
their pupils. The NFER’s approach: 

® Uses data already collected by LEAs/schools; 

o Provides easy-to-grasp quantitative analysis which tells LEAs and schools how 
well they are doing — on a range of key outcome indicators — against norms 
derived from a large number of schools and pupils; 

® Gives expert guidance at each stage of the exercise, from compiling data to inter- 
preting and acting on the results; 

® Helps professional practitioners carry out more focused reviews and self-eval- 
uations, particularly for the purposes of target-setting. 
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D otrody ctiomi 

Traditionally schools have had considerable autonomy in the Netherlands. In the 
recent ‘Quality Law’ it is laid down that schools themselves are responsible for the 
quality of education they provide and the pursuance of a quality policy to ensure 
its improvement. It has also been decreed that all schools develop a system of qual- 
ity assurance. Schools are encouraged to carry out school self-evaluation, without 
providing detailed formats on how to go about this. 

In this context, where a strongly developed public educational support struc- 
ture is a significant factor, there is a broad supply of approaches and tools for school 
self-evaluation. Results of an inventory of these approaches and tools indicated, 
among others, weaknesses in the research technical quality of many of the available 
instruments. Three organizations in the education support structure, the former 
Foundation of Educational Research (SVO), the Foundation for Curriculum 
Development (SLO) and the Institute for Educational Measurement (CITO), joined 
forces in stimulating the development of a more scientifically based instrumentation 
of school self-evaluation in primary education. The result was a collaborative research 
project among CITO, OCTO, the research institute attached to the Faculty of 
Education of the University of Twente, and SLO, aimed at an integrated instru- 
mentation of school self-evaluation for primary schools, called ZERO (self-evaluation 
in primary education). The qualification ‘integrated’ refers to the fact that different 
types and approaches to school self-evaluation and monitoring educational quality 
are being combined. These various approaches also have distinct theoretical and 
disciplinary backgrounds, which are synthesized in this project. Briefly stated, the 
joint instrument consists of a pupil-monitoring part that depends heavily on psy- 
chometric theory and central issues of adaptive instruction (ZEBO-PM), an assessment 
of the educational content covered with central concepts of curriculum planning 
and curriculum evaluation at its background (ZEBO-CC), and the measurement of 
school process indicators, with school effectiveness and school-improvement mod- 
eling as its conceptual background (ZEBO-PI). 

This article provides general information on the total integrated project in the 
Dutch context and more detailed information on one of the three elements: the mea- 
surement of school process indicators (ZEBO-PI). Section two describes the specific 
Dutch history and context of the trend towards school performance evaluation, feed- 
back and improvement. In section three the basic ideas and the strategy behind the 
total project are presented in general, while section four focuses on ZEBO-PI. This 
section describes in more detail the content of the instruments, the feedback system, 
and the support schools receive in interpreting and using the feedback information. 
In section five the results of the studies on the usage of feedback information are 
presented. It provides information on the extent of the use, interpretation problems, 
assimilation of the information, actions undertaken, preferences and tensions. The 
final section contains recommendations for the future. 
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The Dytch context 

In the Netherlands schools have a free choice about the principles on which they 
are based and in the way they organize their teaching (Ministry of Education, Culture 
& Sciences, 1999, Van Oijen in Solomon, 1998, Hendriks, Doolaard, Bosker, 2001). 

This freedom, guaranteed under article twenty-three of the Constitution, results 
in a variety of public and private (but government-dependent) school-types. This 
traditional autonomy in the pedagogical domain, before long, was not matched by 
a similar degree of freedom in the domain of educational finance and the working 
conditions of the teaching force. However, from the 1980s onwards, further decen- 
tralization has been initiated. In 1993 the Ministry of Education and Sciences and 
the organizations of governing bodies of the schools agreed upon more autonomy 
for the schools in the domains of administration and finance, and on decentraliza- 
tion of some tasks to the municipalities. 

On 1 August 1998, the law concerning the quality policy (the Quality Act) 
went into effect. In this law it is laid down that schools themselves are responsible 
for the quality of education they provide and the pursuance of a quality policy to 
ensure its improvement. It has also been decreed that all schools develop a system 
of quality assurance. The law states that the school is required to develop three pol- 
icy documents: a school plan, a school prospectus, and an arrangement for complaints 
(Ministry of Education, Culture and Sciences, 1995), 

The school plan outlines the policy of the school on the quality of education 
and the school improvement activities for the next four years. It has to include the 
policy on educational matters, staffing and internal quality assurance. These are the 
only three compulsory elements. Other topics may be incorporated at the school’s 
own discretion. Next to the function of being an integral, internal policy document, 
the school plan has the function of an accountability document to the inspectorate. 

The school prospectus contains information on the objectives of the school, 
the educational activities and the results achieved. It is a public accounting of schools 
to parents and pupils, it gives them the opportunity to enter a dialogue with the 
school; it enables them to compare schools and it helps them to choose the most 
suitable school. Both the school plan and school prospectus have to be updated every 
four years. 

The school plan, school prospectus and arrangement for complaints have to 
be regarded as a means to an end. Together with other means, they could stimulate 
more systematic quality assurance. They could promote dialogue within the school 
and between the school and the environment. The overall idea is that, more than in 
the past, parents and pupils get the opportunity to affect the quality of education 
(WO, 1998a, 1998b; WVO, 1998). 

During the past few years, in connection with the shift in educational policy, 
external evaluation by the inspectorate shifted from a legalistic supervision to a more 
substantive educational monitoring, based upon evaluation criteria and standards 
developed by the inspectorate itself. In primary education the developments have 
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led to an adapted approach of supervision, consisting of regular supervision and 
integral supervision (Ministry of Education, Culture and Science, 1999). 

In regular supervision the inspectorate studies the school’s results and a num- 
ber of important quality characteristics relevant to the teaching/learning process. In 
addition, the inspectorate evaluates the school plan and the school prospectus. School 
visits also are conducted to supplement the analysis of documents. In general, a 
school-visit within regular supervision lasts for one day. After the visit the main find- 
ings are presented in a report. 

If the regular supervision reveals that problems or the potential for problems 
exist, integral supervision will be carried out. If, on the basis of this integral super- 
vision, improvements are necessary, then the school will be invited to draw up and 
implement a plan of action. Depending on the progress, an intensified tailored super- 
vision will be carried out. 

Primary education has no final examination in the Netherlands. However, 
about 80% of primary schools use a criterion-based assessment test in the final 
grade that is primarily meant to facilitate the choice of one of the secondary school 
types. Primary schools in the Netherlands are used to standardized achievement 
tests, as about 70% of the schools use pupil-monitoring systems for purposes of 
diagnosing the progress of individual pupils. The most frequently used pupil mon- 
itoring system and final-grade-test are developed by CITO, one of the partners in 
ZEBO. 

The development and rationale of ZEBO 

As described in the previous section, the general political climate in the Netherlands 
can be seen as ‘mildly supportive’ of the further development of school self-evalua- 
tion. The providers, available methods and tools for self-evaluation are a reflection 
of the Dutch education support structure on the one hand, and a precursor of the 
‘quality care business’ in the education sector on the other. Both orientations have 
a more ‘social engineering’ background in that they are grounded in empirical ana- 
lytic design and evaluation methodology. Evaluation studies on school self-evaluation 
approaches, funded by the Ministry of Education, noted a general lack of attention 
for the reliability and validity of instruments for school self-evaluation (Cremers- 
van Wees et al., 1996a and b; Hendriks, 2000). 

The initiation of the ZEBO-project is to be seen as a direct consequence of 
these evaluation studies. At the same time, the more empirically analytically inclined 
branch of the Dutch education support structure, particularly CITO and SVO, were 
interested in making their mark on school self-evaluation. The result was a collab- 
orative research project among CITO, OCTO and SLO. The project started in 
September 1995, 

In the most general sense the instrumentation of school self-evaluation in pri- 
mary education should enable schools to assess their own quality. According to the 
school improvement literature, the crux of change resides in the feelings of teachers 
exposed to change (Fullan, 1991). How individuals interpret what change means to 
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them is crucial in the process of transforming change into progress. Therefore, a 
change initiative from within the school is more likely to be successful than imposed 
initiatives because teachers themselves first experience a ‘deficiency’. Regular self- 
evaluation can help schools to detect these deficiencies and to monitor current 
improvement initiatives. Experiencing successes during and as a result of improve- 
ment initiatives results in a sense of mastery, accomplishment and professional 
growth that will influence future change processes and increase the change capaci- 
ty of a school. In this light, the quality assessment can serve three major purposes: 

• To provide basic information about school functioning for quality maintenance 
and school improvement, 

• To provide a starting point for further analysis and diagnosis on specific points 
of school functioning, and 

• To provide a basis for informing relevant audiences about school functioning 
and improvement. 

In order to offer schools the possibility to judge the quality of their education in a 
reliable, valid and efficient manner, the instrumentation of school self-evaluation 
should meet a number of criteria that mainly stem from educational literature: 

• Pupil achievement measures should be the backbone of the instrument. The 
educational outcomes should be measured in such a way that allows a close 
monitoring of pupil progress during the (primary) school career; 

• Input data concerning pupil background characteristics, like socio-economic 
status and initial achievement, should be used to construct ‘value-added’ out- 
come measures; 

• Process indicators, including those concerning the content that is covered, 
should be used to provide ideas and ‘hunches’ as to what accounts for certain 
patterns in value-added outcomes; 

• The use of the instrumentation should not lead to an additional burden the 
schools. If possible, data should be used that are already available at the school, 
such as systems or programmes to store educational results; 

• Users should be enabled to make judgements that are scientifically meaning- 
ful, without complicating these by the use of statistical procedures; 

• A school should be enabled ultimately to make comparisons with other schools 
(Moelands & Ouborg, 1995). 

Furthermore, the instrumentation should be as flexible as possible. If desirable and 
possible, schools should be enabled to use the parts of the instrumentation accord- 
ing to their own information needs. Finally, the instrumentation should be 
user-friendly, lead to results that are easy to interpret, take up less time and make 
use of the possibilities of the communication and information technology. 

Box 1 outlines the possible uses schools can make of the envisaged integrated 
instrument. This is done by listing the type of evaluative conclusions that schools 
are able to draw when the instrument has been used over a period of five years and 
when external reference information is available. 




Prospects, voL XXXI, no. 4, December 2001 




508 



Maria A. Hendriks, Simone Doolaard and Roel J. Bosker 



BOX 1. Using school effectiveness as a knowledge base for self-evaluation in Dutch schools. 

A. Are attainment levels in basic subjects improving or declining over time? (with- 
in schools, within cohorts ‘raw score’ comparisons) 

B. Is the school’s effectiveness improving or declining over time? (as A, using adjust- 
ed, value-added outcomes) 

C. Is the school’s effectiveness up to standard from year to year? (comparative 
analysis on value-added outcomes) 

D. What can be concluded about teacher effectiveness by examining average 
progress-scores of particular cohort grade-level combinations? (within schools, 
between cohorts and between grades analysis on progress scores) 

E. Is the school differentially effective for certain sub-groups of pupils? (e.g. com- 
paring high and low SES-pupils) 

F. How do patterns of differential effectiveness develop over time? 

G. Do amounts of content covered in basic subjects increase or decrease over time? 

H. How do the school’s content-covered levels compare to external referents? (dif- 
ferentiation according to grade-levels and cohorts) 

L Can changes in effectiveness be attributed to variation in content covered? (analy- 
ses per subject, grade-level and cohorts) 

J. Similar questions as G, H and I for other process indicators on school and class- 
room functioning. 

Source: Scheerens, 1995. 

The project was aimed at providing primary schools with instrumentation with 
which they themselves could evaluate the education results, collect data on pupil 
background characteristics and evaluate the curriculum content covered and (other) 
relevant process characteristics (Scheerens, 1995). Although ‘integration’ is one of 
the guidelines of ZEBO, the connections that can be made between certain elements 
of the overall instrument are closer than of other combinations. For instance, achieve- 
ment and prior achievement data have been combined into value-added indicators 
or progress-scores. But there is a looser association between achievement indicators 
and process-indicators, as will be shown in the next paragraph. 

During the project each institute developed their own part of the instrumen- 
tation (i.e. CITO developed the part on output measures and input indicators, SLO 
and CITO the instrumentation on content covered, and OCTO the instrumentation 
on the other process characteristics). The progress was discussed on a regular basis. 
The project finished in August 2000. The progress and the results of the project are 
described in an interim report (Engelen et al., 1997) and a final report (Moelands 
et al., 2000). 

Content^ procedure^ feedback and support 

CONTENT OF ZEBO-PI r 

DU 

The development of ZEBO-PI took place over almost five years. It started in 1995 
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with a study of literature on school effectiveness, school improvement, and perfor- 
mance indicators and ended in August 2000 after a final test in a representative 
sample of 123 schools in the Netherlands. 

The rationale adopted in the choice of process variables for ZEBO-PI was to 
select those variables that, in research, have been shown to be associated with rel- 
atively high ‘value-added’ achievement. From the literature, a set of most relevant 
process factors has been obtained. Next, available instruments to measure process 
indicators and methods for school-diagnosis were compared: 

® To capture the operational core that is usually mentioned in the reviews of 
school effectiveness research; and 

® To get insight into the actual conceptual contents of complex constructs like 
‘educational leadership’ and ‘high expectations for pupils’ achievement’ (see 
also Scheerens &c Bosker, 1997). 

The inventory has led to a broad range of components within factors. At the oper- 
ational level it was revealed that there is little agreement on the substance of key 
factors that are supposed to determine school effectiveness. To check whether they 
would recognize these factors and components in practice, the factors and compo- 
nents were presented to primary school leaders and teachers. 

Then instruments had to be selected or constructed to measure the variables. 
The idea of the instrumentation is self-reporting and judgement by ‘consumers’. So, 
double measures are taken: At the school level, school management provides infor- 
mation using self-reports and are also judged by teachers; and at the classroom level, 
teachers provide information using self-reports and are also judged by their pupils. 
Three instruments, all questionnaires, are developed: one for the school manage- 
ment and the teachers, one for the pupils of grade three (ages 6-7) and one for the 
pupils of grades four through eight (ages 7-12). The questionnaire for pupils of grade 
three is an adapted version of the questionnaire for the pupils of grades four through 
eight. Adaptation includes reduction of the possible answers (‘true’ and ‘not true’ 
instead of ‘true’, ‘a bit true’ and ‘not true’), no negatively formulated statements and 
reduction of the number of items. 

Table 1 shows an overview of the selected process variables and the level at 
which information is collected. 

Box 2 gives examples of items of the pupil as well as the teacher questionnaire 
concerning classroom climate. In the ZEBO-questionnaires, classroom climate is 
related to discipline during lessons, for example: use of clear rules, keeping down 
noise levels, emphasizing that pupils listen well to each other and consequent appli- 
cation of disciplinary rules. 

All instruments meet the criteria of reliability, i.e. the scales are consistent on 
both the individual and the aggregated level. Reliability on the aggregated level is 
an important issue because of the construction of double measures, described in this 
section. 
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Table 1. Selected process variables and level of measurement 



Process variable 


Information collected by 


School level 


School Teachers Pupils 

management 



Achievement orientation/high expectations 


X 


X 


X 


Educational leadership 


X 


X 




Staff development 


X 


X 




Pupil care; measures that enable the realisation 








of inclusive education 


X 


X 




CONSENSUS AND COHESION AMONG STAFF: 








Frequency and content of formal staff meetings 








with the school management 


X 


X 




Frequency and content of informal meetings 








among teachers (co-operation) 


X 


X 




SCHOOL CLIMATE: 








Relationships between staff 


X 


X 




Relationship: the role of the school management 


X 


X 




Workload 


X 


X 




CLASSROOM LEVEL 








Structured instruction 




X 


X 


Adaptive instruction 






X 


Time on task 






X 


Monitoring of pupils’ progress 




X 




Pupil care; special care for high and low achievers 




X 




Classroom climate 




X 


X 


Relationships between pupils 






X 


Support from the teacher and relationship 








between teacher and pupil 






X 



Source: Moelands et aL, 2000. 


BOX 2. Example of items of the pupil questionnaire 


variable ‘Classroom climate’ 


Most pupils listen well when the teacher is talking 


In the classroom I am able to work quietly 


□ true 


□ true 


□ a bit true 


□ a bit true 


□ not true 


□ not true 



Example of items of the teacher questionnaire variable ‘Classroom climate’ 

To what extent do you underwrite the following statements concerning the working climate 
in your classroom? 





Totally 


Slightly 


Slightly 


Totally 


I take care that pupils do not disturb 


disagree 


disagree 


agree 


agree 


each other during their work 


□ 


□ 


□ 


□ 


I use clear classroom rules 


□ 


□ 


□ 


□ 



Source: Moelands et al., 2000. 
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Without further elaboration, conclusions on the validity are mixed and in line 
with other studies in educational effectiveness. Within the groups of respondents 
various variables correlate with each other, so variables measure to a certain extent 
the similar concepts. Exception is achievement orientation/high expectations mea- 
sured by pupils. This variable correlates negatively or not with the other variables 
on pupil level. In general, double measures do not correlate higher with each other 
than with other variables, which means that they measure different concepts or that 
different respondents do not agree on the measured concept. Exception is the mea- 
surement of relationships between the staff measured by school management and 
by teachers. And finally, cognitive pupil progress is only related to time on task (lan- 
guage and arithmetic), adaptive instruction (arithmetic), and classroom climate 
(language). 

PROCEDURE 

At each participating school a contact was designated by means of a letter and a 
‘folder’. After a first short phone call, this person was visited by a research-assistant 
of the university who explained the procedure of the investigation and obtained 
agreement on the implementation of the self-evaluation. To avoid ‘coloured’ respons- 
es as much as possible, it was a prerequisite that, when the pupils fill in the 
questionnaire, their own teacher was not in the class. In order to meet this prereq- 
uisite, participating schools had the choice to switch teachers or to make use of the 
help of the research-assistant. To diminish the burden for the schools, it was decid- 
ed to include in each phase only the pupils of a few grades in the data-collection. 
The questionnaires were sent back to the university, where data-entry, data-analy- 
sis and preparation of the feedback took place. 

FEEDBACK AND SUPPORT 

The main principle of the feedback is based on the comparison of school and class- 
room scores to national averages. Two kinds of feedback reports were prepared: a 
school report for the whole school with information based on the school leader and 
teacher questionnaire, and a classroom report with information based on the pupil 
questionnaire and the teacher questionnaire. The classroom reports were sent to the 
individual teachers; the school report to the school management. Whether these class 
reports are also used on school level depends on the school itself. 

The schools did not receive personal support in interpreting the information. 
Schools could phone the university for help with interpretation, but this happened 
rarely. 

Both school and classroom reports begin with an introduction and suggestions 
about the use of the information. Both reports contain graphs as well as textual 
descriptions of the graphs. In addition, the school report contains information about 
the scores on the individual items of the topics (i.e. the mean scores of the school 
and the deviation within the total group of schools). The school report does not pro- 
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vide information on the individual teachers; the classroom reports do not provide 
information on individual pupils. The way the ZEBO-project feeds back informa- 
tion is illustrated in Figure 1. 

Figure 1. Example graph about the school as a whole and textual description 







□ 



other 

schools 

example 

school 



EXAMPLE TEXTUAL DESCRIPTION 

From the school overview it appears that teachers from the example school in com- 
parison with teachers from other schools are above-average positive about the 
educational leadership at their school. The workload compared with other schools is 
below average and all teachers agree with this (see width of white bar). The scores for 
staff development are somewhat lower than those of other schools. From the school 
overview we can see that the average score on co-operation at the example school is 
very low and that the teachers have divers opinions about this subject (see width of 
white bar). 



Source: Moelands et al., 2000. 



On the basis of Australian experiences, the results are graphically presented in 
so called ‘box and whisker’ plots. The central point of the ‘box and whisker’ plot 
is the mean, i.e. in the white box the mean of the school or classroom, in the black 
box the national mean. The utmost left and right points reflect the tenth and nineti- 
eth percentile, the lower and upper limits of the box the twenty-fifth and seventy-fifth 
percentile. 

When the areas of the school (or classroom) ‘box and whisker’ plot and the 
national ‘box and whisker’ plot between the most extreme points do not overlap, 
there is a clear significant difference between the school (or classroom) mean and 
the national mean. When the boxes are to the right or the left of the national mean, 
there is ‘some’ indication for difference. In all the other cases there are no signifi- 
cant differences between school and national means. The ‘width’ of the bars indicates 
the variability between the teachers (in the school report) or pupils (in the classroom 
report). The wider the bars the more variation exists. 

The textual description contains two kinds of comparisons. In the first place 
the results of the school (or classroom) are being compared with the national results. 
In this case the text serves as an explanation of the plots. Secondly, in the school 
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report the scores of the teacher are being compared with the score of the school 
management. In the classroom report this comparison is also made, although this 
concerns a limited number of scales (i.e. only the scales ‘structured instruction’ and 
‘classroom climate’ of the teacher questionnaire and the scales ‘the lessons’ and 
‘working climate’ of the pupil questionnaire). 

In case there is a statistically significant difference between the scores of the 
teachers and school management in respect of pupils and classroom teacher, this is 
explained in the report. The report only states whether there is a significance and 
not in which direction the difference is, i.e. whether teachers or school management, 
for example, have rated a particular issue higher or lower. Such differences are fre- 
quently encountered and can be seen as a starting point for internal consultation 
and debate. 

In addition, at the request of the schools, in the school report, feedback has 
also been provided at the individual item level. For that, the school average item 
scores are numerically compared with the 80% confidence interval around the nation- 
al mean for each scale. In the explanation, it is mentioned that if the school mean 
on an item is outside this interval, it really differs from the national mean. 

The usage of the feedback 

The utility of the instrumentation and the feedback reports is evaluated in two steps. 
First, together with the feedback reports, all 123 participating schools received an 
evaluation form with questions about the content and usage of the feedback reports 
of the school and the individual classes; 76% of the schools returned the form. 

Secondly, more in-depth, detailed information was obtained by telephone inter- 
views among ten schools. These interviews were conducted in April 2000; i.e. six 
months after sending back the classroom reports and nine months after the school 
report. In most cases the respondents were the school leader or deputy school leader. 
In addition, at one or two schools the interviews took place with the internal sup- 
port teacher.^ The results are presented below. 

Of the schools that returned the evaluation form, 95 % recognized the results 
presented in the school report and found the information fed back (both graphic 
and textual) relevant, complete and easy to interpret. Eight schools were less enthu- 
siastic about the completeness of the feedback. In their eyes the feedback was too 
rough, the conclusions too limited, and the discrepancies between teachers and school 
management not sufficiently explained. In addition, quite a few schools indicated 
that, although the information presented was not always new, the self-evaluation 
and feedback were very valuable in themselves. It has led to more or less structured 
discussions about the aims and direction of the school, the policy and the actions 
for improvement. From the interviews it appears that when analysing the school 
report, most schools focussed their attention first to the graphic presentation, i.e. 
the score of the school in comparison with the national means and the distribution 
of the scores (width of the bar). Afterwards, the scores on individual items were 
analysed. 
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In general, there was less enthusiasm about the classroom reports. Although 
almost 80% of the respondents indicated that the results presented in the classroom 
reports were recognizable and complete, more criticism was mentioned about the 
vagueness and narrowness of the report. Fifteen respondents would like more detailed 
information in the classroom report. In addition, a number of respondents indicat- 
ed that they didn't know if the individual teachers recognized their classes in the 
presented information. As is mentioned before, the classroom reports were sent to 
the individual teachers and it was left to them whether or not to discuss the results 
with the school management or their colleagues. 

For the future, a number of the interviewed respondents indicated that they 
would like to receive less anonymous classroom reports. One school leader would 
like to have a summary of all classroom reports; another school leader is very enthu- 
siastic about the content of the pupil questionnaires and should like to have feedback 
also at the level of the individual pupil. 

The results presented in both school and classroom reports seemed in accor- 
dance with the expectations the respondents had before, although at a number of 
schools the variation of scores (width of the bars) seemed greater than expected. 
When discussing the variation in a team meeting it became clear that in some cases 
it resulted from the different interpretation or misinterpretation of a question. But 
in other cases the opinions differed significantly, although schools weren’t aware of 
this before. 

At some interviewed schools the results of the teacher questionnaire were val- 
idated by classroom visits by the school management. It occurred several times that 
from the questionnaire, which depends on the perceptions of the respondents, it 
appears that the teacher was satisfied with his lessons and classroom management 
while during classroom visits it appears that the school management was not as sat- 
isfied as the teacher. 

Nevertheless, the school management dismissed the possibility that the teach- 
ers filled in the questionnaire with socially desirable responses. According to them 
a better explanation is that the teachers are not always fully aware of their actual 
and desirable teaching performance. In addition, school management suffers from 
‘the dialectics of progress’. Mostly they have a more sophisticated understanding of 
desirable teacher performance and the way and direction in which teacher devel- 
opment should take place in the future. 

Most schools indicate in the evaluation form that they discussed the school 
report with (indicated in declining importance) the team and the school manage- 
ment, the participation council, the school board, the parents’ committee and the 
school counselling service. At a number of schools the report has been discussed 
together with the results of other evaluations, sometimes in parallel, sometimes inte- 
grated. The classroom reports have been or will be discussed in team meetings (most 
frequently mentioned) and in individual meetings between the school leader and the 
teacher involved. In addition, at three schools a combination of both methods will 
be used: the more general aspects will be discussed in a team meeting and the more 
teacher-specific aspects in an individual meeting. 
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For most interviewed schools, ZEBO-PI was not the first self-evaluation instru- 
mentation they experienced. Eight of the ten schools had experience with other 
school self-evaluation instruments. Next to this, at seven schools classroom visits 
took place each year, at one school even three times a year. Mostly these visits are 
carried out by the school management and focus on aspects of instructional behav- 
iour. But classroom visits are also used to judge the effects of school improvement 
programmes and actions (i.e. the implementation of a new language method or the 
effects of professional development). 

The inspectorate had visited eight schools. At seven schools a regular supervi- 
sion took place; one school had an integral supervision. As far as the comparison 
was possible, the respondents mentioned that the results of ZEBO-PI and the inspec- 
torate correspond reasonably. In addition, schools that were supervised at the moment 
the results of ZEBO-PI or other self-evaluations were available indicated that they 
were better prepared for the visit. They were aware of their weak and strong points, 
they themselves had discussed their policy and improvement plans and therefore 
seemed less dependent of the findings of the inspectorate. This led to a more con- 
structive and open discussion with the inspectorate, which was appreciated highly 
by both the schools and the inspectorate. 

All ten schools indicated that they drew up improvement plans; however the 
nurhber of improvement aspects and activities varied. Some schools already had 
improvement plans before participation in ZEBO-PI. In these cases the feedback of 
ZEBO-PI confirmed the plans or gave rise to some extensions or adjustments. Some 
other schools had to start from scratch. At almost all schools the improvement plans 
have been laid down in the school plan, the policy plan or a quality assurance plan. 
To draw up the plans schools used all the information available, i.e., the feedback 
of ZEBO-PI, other school (self-) evaluations and the findings of the inspectorate. 

Mostly, the improvement plans are made for four years and contain the themes 
for improvement. The concrete activities and working methods are being determined 
per school year and elaborated in an annual plan. 

In the plans of all ten schools much attention is paid to the improvement of 
inclusive education and the development towards adaptive instruction. Other often 
mentioned areas of improvement are the implementation of a pupil-monitoring sys- 
tem and the determination of a clear line with regard to subject matter content 
(curriculum quality). 

The schools mention clear objectives and transparency as most important con- 
ditions for a sound school. In advance of the objectives of the evaluation, the way 
the evaluation results will be used and the benefits of the evaluation have to be clear 
and communicated. In the case of ZEBO-PI, where pupils are involved, the objec- 
tives and use should also be discussed with the pupils and their parents. 

Furthermore, schools indicate that it is important that teachers are open-mind- 
ed for feedback and criticism; that they are motivated to realise improvements and 
want to be involved in school policy-making. Self-evaluation requires a climate of 
trust and has to be set in a developmental context. 

Next to this it is important to have instruments of good quality, which are rel- 
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atively easy to administer and give compact feedback also beyond the school level. 
The function of the teacher in the classroom should be central and is the founda- 
tion for improvement. The perception of the function of the teacher should be 
validated by means of class visits and personnel evaluation interviews. As indicat- 
ed before, teachers sometimes seem to be too optimistic about their own function. 

School plans, school prospectuses and the elaboration in annual plans force 
schools to reflect on their own organisation and to perform an analysis of strengths 
and weaknesses. With that, it is important that the school has ownership over the 
evaluation, the evaluation results and improvement actions. Each school has to set 
its own goals and judge its own score. Although external referents on acceptable 
levels of process variables or the judgement of the inspectorate could be helpful, the 
creativity and inside-knowledge of school-teams is required to contemplate partic- 
ular performance patterns. 

Finally, the respondents indicate that school self-evaluation should be carried 
out once every three to five years. By using the same system it will then be easier to 
determine the progress that has been achieved. 

Recommendations for further development 



Based on the evaluation of the instrumentation and the utility and usage of the feed- 
back for further development of ZEBO-PI, the following steps will be taken: 

• Schools will be given responsibility to choose ZEBO-PI, for the usage of the 
instrument and the feedback. ZEBO-PI will be made available for schools in 
a computerized form. Schools can use the instrument at the time they need the 
information, such as when they have to make a new school plan. And they can 
have the output immediately. 

• The flexibility of the instrument will be enlarged. Schools might want to use 
the pupil questionnaire more often than the other questionnaires. They might 
use only parts of the questionnaires concerning a specific issue. 

• The flexibility of the output and the feedback will be enlarged. Instead of giv- 
ing a standard global graphical impression and item scores in tables, schools 
can fulfil their own wishes; for some scales only the global output; for others 
item scores or even individual scores. 

• Norm-referenced tables (on the basis of percentiles) of the actual performance 
of a representative reference group of Dutch primary schools will be included. 
By comparing the mean score of the school with the percentile of Dutch schools 
with the same or lower score, the school will get more ‘sharp’ information. 

• The support structure will be altered. Next to making the instrument avail- 
able, the university is going to co-operate with a school counselling service, 
which can support the schools that use ZEBO-PI. Furthermore, the manual 
will be extended with more information about the interpretation of feedback, 
suggestions for further activities either for school improvement or for in-depth 
analysis, comparisons with and the use of other evaluation methods and address- 
es for further information and help. 
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Another possibility for an even more flexible self-evaluation instrument could be to 
make a distinction between basic monitoring and signalling functions on the one 
hand and on the other additional instruments for further analysis and diagnosis. 

Finally, to get more empirical information on the use of performance feedback 
the university received a grant to carry out a quasi-experimental study. In this study 
a control group of seventy-five primary schools will receive feedback only on stu- 
dent progress, whereas the experimental group of seventy-five schools will also obtain 
information on process indicators. In addition, an in-depth observational study will 
be carried out to identify characteristics of schools (both in the experimental and 
control group) that manage to increase their performances most in the period stud- 
ied (a two-year interval). This observational study will be designed using promising 
hypotheses derived from the available literature on optimizing performance feed- 
back and evaluation. 

Note 

1. An internal support teacher assists other teachers in developing and applying instruc- 
tion methods that allow for dealing appropriately with differences between children in 
ability, behaviour, etc. This post was created after a new policy in Dutch education on 
integration of pupils from special and regular education (the so-called inclusive educa- 
tion) was implemented. Internal support teachers are classroom and teacher-oriented 
in contrast to remedial teachers who focus on assisting individual pupils with learning 
problems. 
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SCHOOL autonomy and evaluation 
ANALYSIS OF SCHOOLS' 
FUNCTIONING, ASSESSMENT 
AND SELF-ASSESSMENT; 
PRIMARY SCHOOLS IN GENEVA 

Bernard Favre 



I ntroduction 

What relationship is there between the assessment of schools’ performance and a 
sociological analysis of the way they function? That is the question we intend to 
address here, on the basis of the work we have been carrying out in Geneva on pri- 
mary schools as autonomous units and local educational communities, which has 
enabled us to study their relationships with the various parties involved on a daily 
basis in the schooling of children.' 

The assessment of schools is based on the assumption that the school as such 
plays a crucial role in the realization of the objectives of the education system — that 
is, in the way pupils learn, in the broadest sense of the term. It assumes that the 
school enjoys a certain degree of autonomy regarding the resources it employs to 
achieve these aims and also that a teacher’s action only becomes fully effective when 
he or she is an integral component of a team and of an educational community. 

In a centralized education system, however, and particularly in primary edu- 
cation where autonomy — if it exists at all — means above all that of the class teacher, 
it is difficult to justify assessment of the school itself, since the unit of action is the 
body of interactions between the teacher and his or her pupils. The school itself does 
not appear to constitute a specific system of action affecting the extent to which 
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pupils meet objectives in the course of, or by the end of, primary education. What 
is taken for granted in systems where schools have a large degree of autonomy poses 
an immediate problem in a centralized education system: it is difficult to see the rel- 
evance of assessment of the school's contribution to the performance levels of its 
pupils when there appears to be little that is ‘real’ about the school, apart from its 
material aspects (such as the school premises). In this connection, Bressoux (1993) 
showed that the individual teacher was crucial to the performance of pupils learn- 
ing to read in French primary schools. To take the analysis further, one could ask 
whether the school itself plays a more important role when it has more autonomy 
and places greater emphasis on co-operation between teachers. This is the first ques- 
tion to which an institutional and organizational analysis of schools should be able 
to provide some answers. 

A second question should also be addressed: the most difficult task in assess- 
ing the effectiveness of training is that of ‘designating, selecting and using a certain 
number of indicators designed to facilitate a precise understanding of the particu- 
lar process being assessed' (Barbier, 1985, p. 196). If one is assessing a school as the 
‘action system’ concerned, it does not seem feasible to base oneself solely on what- 
ever practical knowledge one might have; ideally the assessor should have a more 
precise understanding of the way the school functions, which means that the fac- 
tors involved in the realization of the objectives pursued must be identified beforehand, 
since the assessment is based on the relationship between the different elements of 
the ‘action system' and the objectives pursued. 

Many studies have already been carried out on the way schools function,^ but 
the relevance of these studies, taking into account the specific organizational con- 
text and level of teaching concerned, has yet to be demonstrated, all the more so if 
performance indicators play a key role in the assessment of schools. If these indi- 
cators are to be of use to the actors themselves, the processes at issue in the training 
leading up to the results need to be elucidated. Informing a school that the perfor- 
mance of its pupils is — all other things being equal — considerably lower than that 
of a neighbouring school does not help it to improve if the factors involved have 
not been taken into account. One might object that assessment requires the con- 
struction of a system of indicators, and not simply performance indicators. For this, 
however, one would need to be able to justify this construction and then investigate 
why in some cases this system of indicators appears to provide insufficient expla- 
nation. Thus, the assessment of a school assumes that the way it functions has been 
analysed and, in many cases, raises new questions in that respect. In other words, 
analysis of how a school functions precedes and follows its assessment. 

Finally, and it is this third hypothesis that this paper seeks to illustrate — if self- 
assessment of a school is to contribute to its improvement, it cannot content itself 
with such rough and ready indicators as the degree of satisfaction of the actors them- 
selves, the school’s atmosphere, or the level of co-operation between teachers;^ it 
must also make use of analytical tools and knowledge of the way the school func- 
tions that are closely linked to the tools and conceptual frameworks used by 
organizational sociologists. g 
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Primary schools in Geneva: the system as a whole 



Geneva has a centralized education system in which, for a long time, co-ordination 
between teachers was ensured by obliging all schools to adopt the same curricula, 
teaching methods and timetables. The application of these rules was monitored by 
inspectors, each of whom was responsible for an average of seventy to eighty class- 
es; some of these inspectors quite openly define themselves today as heads of districts, 
in other words groups of several schools. This shows that the school as such appears 
to count for very little. 

Co-ordination was also based on the fact that teachers had received the same 
training, focusing primarily on teaching practices and acquired during periods of 
practical training as well as through instruction in methods based on the prepara- 
tion of model lessons that teachers were supposed to reproduce in their classrooms. 
Because of this, teaching practices were automatically homogenous, and co-ordi- 
nation between teachers was scarcely necessary, as all of them complied with central 
directives. 

Since the early 1970s, while teachers still follow the same training and are part 
of the same administrative ‘mould’, the trend has been towards greater flexibility of 
curricula and study programmes, which are gradually becoming more goal-orient- 
ed and less content-oriented. Similarly, there has been a trend towards greater 
flexibility in teaching methods. Guidelines are issued to all teachers but, as they are 
increasingly based on the research of teaching specialists in the various disciplines, 
teachers themselves are being invited to participate in discussion and research. 

This ‘liberalization’ has not, however, translated into greater autonomy for 
schools in Geneva. As far as administrative aspects are concerned, for example, 
schools have no say in the hiring of teachers, are obliged to accept pupils whose 
families live within the school’s catchment area (parents are not free to choose the 
school they wish their child to attend), do not have their own budget and have very 
limited power in the daily management of the school (officially the headteachers 
have administrative — and in some cases disciplinary — powers only). 

This situation is changing. With the reform of the primary-school system/ a 
team of teachers in the schools are now asked to draw up a plan de travail (work 
plan) then a projet d^ecole (a school’s ‘mission statement’) of four-year teaching 
cycles, involving mainly its management.^ 

It is expected that these measures will result in schools becoming more 
autonomous and, in particular, in the introduction of self-assessment and peer mon- 
itoring rather than supervision by hierarchical superiors. To a certain degree, the 
autonomy of schools has been imposed by administrative decree, in the hope that 
it will oblige teachers to work together. It is thus in a context of ‘dogmatic innova- 
tion’, in the sense defined by Alter (2000), that the issue of assessment and 
self-assessment is gradually being addressed in primary schools. One could even 
form the hypothesis that the autonomy of schools — and thus their specific individ- 
ual responsibility — is being encouraged in order to confront the gradual fragmentation 
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of teaching methods, the difficulty of controlling this fragmentation in a centralized 
manner, and the multiplicity of approaches adopted by the teachers themselves. 

The investigations to be described below are being conducted in this context 
of transition and great uncertainty at the level of the system: the aim is to find out 
more about the practical conditions that need to be changed. What are the prima- 
ry schools in Geneva on which the changes have been imposed really like? First, they 
differ greatly in size; while some have two or three classes, others have over thirty. 
Size depends on location: schools in the city are generally large, while those in coun- 
try areas tend to be smaller. There is a whole range of gradations in between, since 
the rural areas around Geneva have gradually turned into residential areas which 
are growing rapidly and attracting mainly middle- and upper-class families. Schools 
located in solidly working-class areas, whether in the city or its suburbs, have a high 
percentage of non-Swiss pupils whose mother-tongue is not French,^ while those 
located in residential or rural areas have only a minority of children from working- 
class backgrounds. This diversity, which is bound up with the urban environment 
and the strict application of the ‘catchment area’ principle to identify the school a 
child should attend^ makes it difficult to compare one school with another, even in 
a strongly centralized system. It is compounded, moreover, by the diversity of teach- 
ers’ ways of dealing with it, adapting their teaching to local conditions or simply 
developing their own informal strategies in the many ‘grey areas’ not covered by the 
centralized regulatory system. 

This is where our analysis of the functioning of primary schools in Geneva comes 
in. We are seeking to establish in as much detail as possible what each school does in 
the areas of freedom left to it by a heavily centralized system. In terms of assessment, 
and at a time when primary schools are undergoing a reform process in which more 
importance is being given to local initiatives, such an approach has two advantages: 
® It should make it possible to identify the specific objectives pursued by the 
teams of teachers in each school, considered by them to be criteria for assess- 
ing the quality of their work, and later what indicators teachers use in their 
own ‘spontaneous’ assessments. 

® It could also provide decision-makers with information on the ‘assimilation 
mechanisms’ developed by schools over time, which could be considered as the 
foundations on which to base the changes called for by the centre. In other 
words, how does the new way of operating wanted of schools fit into — or 
obstruct — the areas of autonomy they have appropriated? What initiatives have 
schools taken so far to adapt to local conditions, and on what local innovations 
(regarding curriculum, organization, teaching strategies) could reform be based? 

Ten primary schools in Geneva: 
organizational analysis 

In order to start answering these questions, we selected ten primary schools, ensur- 
ing that as far as possible they covered the range of characteristics observed in respect 
of size, geographical location (city, suburb or rural area, type of commune or neigh- 
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bourhood), social background and national origin of pupils, and whether the school 
was one of those taking part in the reform.^ 

We adopted what has now become the traditional framework of organizational 
sociology (the French school in particular, but making considerable use of British 
research),^ focusing in particular on the role of those involved, the agreements and 
conventions that allow them to live and work together, the processes governing 
inter-group relations within the school and the school’s relations with its environ- 
ment. Within this framework, analysing the way a school functions involves: 

® Identifying the historical background of events, shared practices and arrange- 
ments that have resulted in shared values and a shared culture, with which all 
teachers can identify. 

® Analysing the compromises that teachers make among themselves, when each 
one (or each sub-group) invokes different principles to affirm the legitimacy 
of their approach. We are basing ourselves here on the hypothesis advanced 
by Derouet (1992) and the ‘conventions’ school — see in particular Boltanski 
and Thevenot (1991) — according to which actors are no longer guided in mod- 
ern individualist societies by a single principle of justice, but justify their actions 
by invoking concurrent principles of justice, and therefore definitions that are 
not those of the ‘accepted’ school of thought. 

® Identifying the processes that actors and groups of actors implement in their 
relations with each other and in relations between the school and its environ- 
ment: one school will take the attitude, vis-a-vis parents, for example, of fighting 
its corner whereas another will be more receptive and work on various forms 
of co-operation with parents, including ways of helping children learn. 

• Identifying the many networks to which the school and its members belong 
(the most widely accepted educational movement, social services, training col- 
lege lecturers, researchers at the Faculty of Psychology and Educational Sciences, 
University of Geneva). Allegiance to one or other of these networks can result 
in different strategies, alliances and sympathies or, on the contrary, in certain 
forms of isolation. 

With regard to methodology, we compiled, for each of the schools analysed, infor- 
mation concerning the teaching staff (average age, replacement rates, training, 
professional background, etc.), pupils (socio-professional category and national ori- 
gin of families, trends in pupil numbers), the neighbourhood or commune in which 
the school was located (type of environment; demographic trends; cultural resources; 
whether or not there was a parents’ association),*’ the school’s relations with insti- 
tutions providing psychological or social support (child-guidance centres, social 
services, local associations, out-of-school or non-formal educational services, etc.). 
To understand the way the schools functioned, we carried out a series of interviews'- 
with most teachers, particularly the headteacher.’^ These interviews were organized 
around some main themes: teachers’ professional backgrounds (training, previous 
experience, personal experience since arriving at the school); how they co-operated 
and communicated with colleagues; the role played in the school by the headteacher 
and the inspector; their expectations in this area; their view of the teacher’s role; 
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their image of the pupils in their class; their definition of when a pupil is struggling 
or failing to keep up; and their personal reactions with regard to the broad lines of 
the reform proposed by the Directorate of Primary Education. Headteachers were, 
in particular, asked to describe the role they wished to play in their schools, what 
they thought their colleagues expected of them, and their relations with the inspec- 
tor and headteachers in other schools. We sometimes met the school caretaker and 
groups of pupils who were invited to give their views on their life at school, their 
relations with teachers, the way lessons were conducted, their relations with fellow 
pupils, etc. We also asked to sit in on certain classes, with the agreement of the teach- 
ers concerned.*'^ We attended staff meetings (conseil d'ecole)^ and spent as much time 
as possible in the staffrooms, school corridors and playgrounds, whenever that could 
be done without disturbing the day-to-day running of the school or embarrassing 
teachers too much. 

We should also make it clear that the schools we studied did not take part on 
a voluntary basis, since we chose them in order to have good examples of the dif- 
ferent sizes, geographical locations and socio-demographic characteristics already 
identified above. However, we did negotiate a kind of ‘contract’ with the teachers 
of the schools in which we worked. The understanding was that they would open 
up their school to us, agree to reply to our questions and provide us with all the 
information we required, while we undertook to show them the results of our research 
and record their comments. In this way, we provided teachers with data on their 
school which they were then free to use for their school’s benefit, as they saw fit. 

Some results and how they tie in with 
an assessment of schools 

It would be impossible to report here on all the findings of our research, which 
focused on quality rather than quantity and thus provided a very broad range of 
information. We shall simply identify a number of factors and show how they can 
be used in an assessment of schools: 

• The history of each individual school and how it fits into an institutional sys- 
tem, which in turn has its own history, strongly affects the school’s ‘culture’ 
and the way it operates. 

• Although a centralized education system weakens the role of management at 
the level of individual schools, local ‘reinterpretations’ are a key factor in the 
emergence of innovative trends. 

• Although traditional methods of regulating teaching practices have resulted in 
an individualistic approach in the classroom that has often been deplored, each 
school develops its own type of co-operation among the different teachers. 

HOW THE CULTURE OF A SCHOOL DEVELOPS OVER TIME 

In most schools with more than ten classes the teachers speak of a cold, stiff atmos- 
phere existing in the school in the past (in general seen to refer to a period before the 
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1970s), characterized by reserved relations between colleagues, with widespread use 
of the formal form of address {vous)^ and an authoritarian headteacher who stood on 
his/her dignity and insisted on discipline. But the 1970s brought big changes. Relations 
became warmer, the familiar form of address {tu) between teachers became wide- 
spread and they began to take meals together and celebrate each other’s birthdays. 

Some schools moved further than others in this direction, under the leadership 
of particularly charismatic headteachers. In one large school, for example, at the 
start of the 1970s, the intake of a younger crop of teaching staff coincided with the 
arrival of a headteacher who had been trained in Freinet techniques and was very 
keen to create a warm and friendly atmosphere in his school. He organized regular 
ski trips for teachers, a film club and meals out in local restaurants once a month. 
Some teachers went on holiday together and close links were established between 
their families (it was not uncommon, for example, for colleagues to be godparents 
of each other’s children). Teachers would meet frequently in the staff room, which 
was always a lively and friendly place. The school was known for the welcome it 
extended to new teachers, substitute teachers and visitors (the researchers them- 
selves can vouch for this). 

By no means all the schools we analysed had the same atmosphere, but the 
teachers in most of the schools spoke of the quality of relations between adults, 
either deploring the unpleasant atmosphere in their own schools, saying it had dete- 
riorated over the years, or emphasizing the importance of this aspect of communal 
life: a good school was, first of all, one in which everyday relations with colleagues 
were pleasant and friendly. 

As a general rule, however, this focus on the quality of relations between teach- 
ers does not lead to systematic discussions and arrangements for working together 
or — unless very indirectly — to a ‘communal’ style of relations with pupils. To some 
extent — certainly in the case of the school referred to above — relations in the staff 
room are governed by a ‘system of love’,^^ and in the event of problems or conflict 
the principles of justice in the family are invoked (see Boltanski & Thevenot, 1991; 
Derouet, 1992) (the headteacher being seen as a father figure). However, as far as 
relations and work with pupils are concerned, it is still the ‘system of justice’ that 
operates, a form of civil justice, where the focus is less on adapting teaching to the 
specific characteristics of each pupil, or ‘individual teaching’, and more on the qual- 
ity of the class teaching, on equal treatment of all and on non-discrimination. The 
nature of the school’s intake (primarily working class or lower-middle class) is rarely 
mentioned, except to say how it has changed over the years. What is most com- 
monly stressed is that there can only be close relations between colleagues if the 
professional abilities of each teacher are demonstrated and recognized. ‘Dead wood’ 
would not be tolerated in this staff room for long; it would detract from the quali- 
ty of interpersonal relationships. 

To a large extent, the situation in this school is indicative of a more general 
situation; the definition of a ‘good school’ in the first place as one whose teachers 
get on well, where relations are warm and friendly and conflict is relatively rare (and 
frowned upon when it does occur), has gained ground since the start of the 1970s. 
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It is the one feature which most clearly distinguishes one school from another. If a 
teacher likes a particular school and has chosen to stay there, it is because he or she 
feels comfortable there, has been made to feel welcome and was given support in 
his or her first years of teaching. Untenured teachers working in several different 
schools (for example, art, physical education and craft teachers) adopt the same 
kind of criteria to assess the various schools where they work. This importance 
attached to the quality of relations in the staff room is probably related to the fact 
that teachers in Geneva have fixed posts and have virtually life tenure, except in the 
event of a serious problem or a specific request on their part. 

The reason we have stressed the widespread feeling that a good school is a har- 
monious community of adults is because it is not necessarily conducive to the 
introduction of team work, as part of an efficient and autonomous process, for it 
implies, to a certain extent, a desire to avoid conflict. And yet conflict is probably 
an essential part of teamwork and of comparisons between different teaching prac- 
tices. Many teachers are wary of drawing such comparisons. 

MANAGEMENT IN PRIMARY SCHOOLS 

Formal responsibility for management in primary schools is exercised by a head- 
teacher whose official function is essentially administrative.'^ However, in reality, 
headteachers play an important role in ensuring consensus and a good working 
atmosphere. Replies to a questionnaire distributed to a representative sample of 
teachers in Geneva'^ confirm this observation, enabling us to establish the follow- 
ing order of priority in the roles assigned to the headteacher: 

• First, responsibility for representing the school in relations with the educational 
and local authorities. 

® Then, close behind, responsibility for creating a good atmosphere in the school 
and for resolving tensions (the headteacher is usually the authority on moral 
and ethical issues), and for ‘maintaining good relations with parents' (mainly 
via the parents’ association, but once again it is interesting to note that it is the 
quality of relations that is being referred to). 

® Tasks relating to the exercise of authority on teaching methods were only rarely 
attributed to the headteacher (the questionnaire categories were as follows: 
‘concern for the quality of teaching’, ‘floating new ideas relating to teaching 
methods’, ‘co-ordinating the introduction of new teaching methods’). 

This last finding is of particular importance in a context of greater autonomy for 
individual schools, although it is quite in keeping with the formal definition of the 
role of the headteacher. Only 21% of the teachers who replied to the questionnaire 
considered that the headteacher should co-ordinate the introduction of new teach- 
ing methods and 19% that he or she float new ideas relating to teaching methods. 
Not all teachers refused to assign these roles to the head, but a great deal of reluc- 
tance was expressed, a quarter of teachers even being of the view that the task of 
co-ordinating the introduction of new teaching methods was definitely not the job 
of the headteacher. 




Prospects'; vol. XXXI, no. 4, December 2001 




Primary schools in Geneva 



527 



These findings confirm the view outlined above that the teaching community 
is primarily a function of interpersonal relations, and the main role of the head- 
teacher inside the school is to maintain the quality of these relations. However, the 
different roles of the headteacher vary from school to school. In other words, in 
each of the schools studied the head and the teaching staff ‘invent’ a way of acting 
out this role. Earlier on we described one headteacher as a charismatic leader and 
‘father figure’ who shied away from any involvement in teaching methods. There is 
also the ‘lightning conductor’ type, who tries to maintain a harmonious atmosphere 
by defusing over-enthusiastic calls for change or by acting as a shield between staff 
and demanding inspectors or interfering parents. At the other extreme one also — 
occasionally — comes across headteachers who take a leading role in the organization 
and development of teaching activities and try to carry their staff with them in inno- 
vative initiatives. 

While interpretations of this role vary from school to school, they all share a 
certain wariness of any tendency for the head to impose his or her authority in the 
specific area of teaching methods. This is each individual teacher’s ‘secret garden’, 
and if the head is accepted as having any authority in this area it is on an informal 
basis and must be related to his or her specific skills in a particular field (discipline, 
techniques, methodology), acknowledged by the teachers but not subject to official 
recognition. 

When addressing this matter of authority, the role played by school inspectors 
also needs to be studied; it is no longer one of checking that the teaching practices 
of individual teachers comply with the models established by the system, but rather 
that of ‘firemen’ called upon to intervene in the event of serious conflict between 
teachers or with parents. However, inspectors also try to ‘reinvent’ their roles, most 
often by encouraging innovative initiatives, regulating the appointment of teachers 
so as to strengthen a new team or ensure a balance between teachers with new ideas 
and more traditional teachers; in other cases, especially where there is an ‘innova- 
tive’ school in an inspector’s district, they will support initiatives and act as mediator 
when the team is unable to solve certain difficulties itself, while leaving teachers as 
broad a margin of manoeuvre as possible.’*^ 

METHODS OF CO-OPERATION BETWEEN TEACHERS 

If there is one theme that is constantly harped upon when normative approaches to 
teaching are invoked, it is that of individualism: the classroom being seen as off-lim- 
its. Once the door has been closed, the teacher alone is in charge, and any attempt 
at scrutiny by an outsider is strongly resisted. Our organizational analysis calls this 
truism into question or at least requires a less categorical stand on the matter. 

As we saw earlier, in a centralized education system arrangements for the reg- 
ulation of teachers’ work presuppose that they operate in isolation, the way they 
work being dictated by the curriculum, the textbooks and the teaching methods 
adopted, and co-ordination between the different levels ensured by all teachers com- 
plying with the rules handed down from above. In this sense, the individualism of 
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the teacher is a product of the system. On an informal level, however, there is a great 
deal of interaction, although social relations between teachers vary considerably 
from one school to another; in some schools interaction is limited to a minimum of 
civility while in others teachers outside the classroom form a real community, dis- 
cussing not only pupils but also problems encountered in the classroom and teaching 
approaches, but always informally. Over the last twenty years classes have had to 
open up to special needs teachers, who help those pupils with the most difficulties, 
and specialized teachers (craft, physical education, music, etc.). The number of class- 
es taught by two teachers^^ working together has continued to rise, accounting in 
1998 for approximately 150 out of a total of 1,400 classes. Teaching in tandem is 
not strictly speaking a form of ‘team teaching’, but it does imply continuous co-ordi- 
nation between the teachers and two different approaches to pupils. Classes have 
also opened up to parents, and a far from recent study^® showed that almost one- 
third of teachers (especially at elementary level) organized open classes for parents. 

In the schools we studied, we observed a great variety of forms of collabora- 
tion. In some schools it was kept to a minimum, work in the classroom remaining 
very much a solitary task, albeit with the occasional exchange of documents, ideas 
and information concerning pupils. At the other extreme, co-operation would involve 
teaching practices themselves; for example, teachers would prepare certain teach- 
ing sequences together, compare the results obtained and change their approach 
accordingly. We are thus able not only to construct a typology of ways in which 
authority is exercised, but also a typology of forms of collaboration between indi- 
vidual teachers. 

Organizational analysis as an aid to 
the self-assessment of schools 

In the previous section we focused on aspects of the ways schools function that are 
often used as indicators of their quality: the way authority is exercised, methods of 
collaboration between teachers, the extent of agreement between them, their view 
of the teaching profession, etc. While organizational analysis shows that these fac- 
tors, within each school, constitute a system or are closely related, it also shows that 
they are neither static nor easy to control. The forms they take in each specific school 
are the visible expression of social processes that are highly temporal in nature and 
of complex interactions among teachers and between the school and its environ- 
ment (not only the education system as a whole, but also the local environment, 
encompassing parents, local authorities, etc.). The balance between all of these ele- 
ments is sometimes fragile and interactions can often appear to be ‘blocked’, 
particularly when tension between the various elements is high. Seen from the out- 
side, it is easy to explain this situation in terms of resistance to change or to denounce 
the inertia or lack of co-operation of some actors. In our view, if organizational 
analysis is to contribute to a type of assessment from which lessons can be learned, 
the actors’ logical processes and strategies must be analysed from within and a com- 
prehensive sociological approach adopted.^^ We shall illustrate these rather abstract 
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theoretical comments with two examples of the way schools function that were 
observed in our case studies. 

DIFFICULTIES IN ABANDONING THE ‘HAPPY FAMILY’ APPROACH 

This particular school is located in the countryside near Geneva and today has around 
fifteen classes. Less than twenty years ago the commune was inhabited mainly by 
farmers and market gardeners. There were only three or four classes, and each teacher 
was teaching pupils of two or three different grades. Relations between teachers and 
classes were very informal, adapted to life in a small school, as were relations with 
the local authorities. However, in the space of a few years the population expand- 
ed considerably, and the commune became mainly residential, with a majority of 
middle- and upper-class families. Instead of three classes there were now fifteen, and 
it had become clear that the forms of management and communication between 
teachers that had been perfectly suitable until then no longer worked. The same 
headteacher remained, however, and, like some of the new teachers who came to 
join the staff, favoured a liberal, ‘happy family’ type of management approach. 
However, other members of the team felt that this was not enough and pushed for 
a more formalistic approach to rule-making. At the same time, a parents’ associa- 
tion was created which turned out to be particularly aggressive and demanding. At 
the time we conducted our analysis, the teaching staff seemed to be divided as to 
which strategies to adopt for relations both inside and outside the school. Seen from 
the outside it looked like deadlock. In fact, under the surface, a lot was going on 
and the resistance to change was the expression of a conflict that teachers were 
unable to put into words. 

The only way in which the school’s functioning could adapt to the new situa- 
tion in the long run would be for the teachers to realize what was happening and 
work through the latent conflict. However, this in turn would beg other questions, 
for example: does the school have the skills necessary to operate a new regulatory 
system (for example management skills, official formulation of relations between 
teachers).^ Is the inspector able to understand what is happening and take effective 
action.^ Could the status quo not be maintained under the existing centralized man- 
agement system as, after all, the school is still functioning and all the teachers are 
working to the best of their abilities and observing the rules? 

This example highlights the importance, for the process of assessment and self- 
assessment, of analytical tools that enable those involved to grasp their situation in 
all its complexity and to translate the issues into ideas leading to the design and 
introduction of new regulatory systems. 

WHEN PUPILS RESIST TEACHERS’ STRATEGIES 



This example concerns a school where some teachers decided in 1995 to draw up a 
projet d'ecole (a ‘mission statement’ for the school) and to make it part of the pri- 
mary school reform. The school is a large, urban school with a majority of 
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working-class pupils (almost 60% of pupils come from working-class or non-Swiss 
families). The idea of drawing up a projet d'ecole was strongly supported by a small 
group of specialist teachers (crafts, music, physical education). The cost-cutting pol- 
icy operating at the time made these teachers feel that their posts were seriously 
threatened, and by actively participating in drawing up the projet d'ecole they were 
able to draw attention to the importance of their work, especially in a working-class 
school. They believed that the subjects they taught could fire the enthusiasm of pupils 
whose background did not necessarily incline them to pursue intellectual activities. 
An important part of the projet d'ecole was therefore the creation of decompart- 
mentalized workshops,^^ each attended by a small number of pupils. Participation 
in these workshops was conceived as a kind of stepping stone to more scholarly 
activities. The idea was that children would begin to enjoy going to school more 
because of these workshops and thus be more highly motivated when it came to aca- 
demic subjects, which they tended to reject. 

The idea was welcomed enthusiastically by the teachers of more academic dis- 
ciplines, several of whom supported the idea of teaching teams and the active school. 
However, when we visited the school the teachers voiced concerns about the organi- 
zational problems involved in setting up the decompartmentalized workshops, which 
took up most of the time available for team-working. Moreover, the projet d'ecole 
did not seem to have increased pupils’ interest in French and mathematics, although 
it had been designed primarily as a means of combating underachievement at school. 

This example shows a group of teachers apparently using the projet d’ecole to 
serve their own interests by highlighting their contribution to pupils’ learning. This 
kind of situation is very familiar to organization analysts, who have no problem in 
defining an organization as a set of mechanisms which enables actors or groups of 
actors pursuing their own goals to nevertheless achieve the overall goals of the orga- 
nization. We should note, however, that in this case it was not simply a matter of 
private goals:^^ this group of teachers justified its action on the basis of principles 
considered to be generally valid in schools today, and which are shared by their col- 
leagues. The difficulty arises when the problem is posed in terms of effective learning 
for pupils from working-class backgrounds. What may seem right to teachers — 
themselves middle class — who subscribe to middle-class theories of childhood and 
to learning methods which focus on a child’s autonomy and fulfilment, is not nec- 
essarily appropriate for children from working-class backgrounds.^'’ 

In this case, an analysis which focuses on the actors themselves identifies the 
type of reasoning that guides them: in this analysis the actors are taken seriously 
and encouraged to question their middle-class preconceptions. This leads to a form 
of ‘decentring’ encouraging them to consider exactly what methods of academic 
learning are suitable for children of working-class backgrounds and to question 
whether, by motivating these children with familiar activities, they are not encour- 
aging their tendency to shun academic learning processes, and whether such processes 
should be linked in different ways to activities exploiting their manual skills. It is 
not certain, in any event, that performance indicators alone would have enabled this 
team of teachers to reconsider the way it operated. 
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INSTITUTIONAL REGULATION AND ASSESSMENT OF SCHOOLS 

An organizational analysis of schools must necessarily take into account the dynam- 
ics of the relationship between a school and its environment. In the examples above, 
we saw the pressure that can be exerted on schools and the way they function by 
system-wide standards, pupils’ families, and the mind-set and ideas of the social 
groups to which the teachers belong. We shall now briefly show how the messages 
conveyed by the education authorities influence schools’ internal dynamics. 

Among the schools we analysed, those that have accepted the reform process 
all subscribe — in one way or another — to the idea of ‘teaching teams’. Teaching 
teams have a long history in Geneva; they expanded fairly rapidly from the end of 
the 1970s onwards, and since then have had a chequered relationship with the school 
authorities. The Directorate of Primary Education encouraged the creation of such 
teams under pressure from the teachers’ union, but these — militant — teams saw 
themselves as active critics of a way of functioning considered to be excessively 
bureaucratic and of teaching methods they judged to be too authoritarian or rigid. 
In some cases, they disseminated the ‘child-centred teaching’ theories of the active 
school, which were by no means universally approved by teachers and parents. These 
teaching teams met with varying degrees of success, but the primary-school educa- 
tion reform launched in 1995 gave them a new role, focusing them very firmly on 
effectiveness and the ‘mission statement’ — on the model of industry. Many teach- 
ers, nevertheless, had the impression that one particular trend or ideology had thereby 
been given official sanction, and this launched a new version of the debate between 
the ‘traditionalists’ and ‘modernists’, reviving painful memories in certain schools. 

All this calls into question the objectives of the primary-school reform and the 
underlying meaning of the exhortations to teachers to be more independent and pro- 
fessional, In other words, in a traditionally centralizing system dominated thus far 
by a bureaucratic regulatory system, messages from the education authorities, even 
when intended to call into question former modes of operation, are still perceived 
as stemming from the same bureaucratic approach. 

Organizational analysis has to identify the often tortuous routes travelled by 
messages as they wend their way from the centre to the periphery, and the contra- 
dictory interpretations, ambiguities and, in some cases, crises of confidence that can 
ensue. It thus highlights the close links between changes at the local level and changes 
in the regulatory systems as a whole. The work of clarification that is thus made 
necessary is a key factor in determining the effectiveness of any assessment or self- 
assessment of schools. 

By way of conclusion 

COMING TO GRIPS WITH THE COMPLEXITY OF ORGANIZATIONS 

In our view, any purely evaluative approach towards schools is bound to be caught 
up, to varying degrees, by the issue of ends and means: what means does a school 
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employ to achieve the levels of performance required of it? Are these means the most 
suitable and the most effective? In our view, one of the most useful contributions 
of the organizational analysis of schools is that it shows that the way they operate 
in real life does not in fact obey the linear logic of ends and means. Collective action 
does not follow this type of logic. The path linking the means to the end wends its 
way through a complex system of interactions which includes pupils as well as teach- 
ers. While teaching methods, the situations in which pupils find themselves, the 
problems they have to solve and the formal machinery promoting co-operation 
between teachers all constitute necessary forms of mediation, their effectiveness is 
always dependent on the way they are used and the meaning they are given by the 
actors involved. 

For this reason we think it would help if the actors were to use certain orga- 
nizational analysis tools for themselves. Crozier (2000), in line with Argyris, 
distinguishes ‘utilizable knowledge’ from ‘applicable knowledge’: ‘All knowledge 
in the social sciences,’ he says, ‘that increases our understanding^^ of the behaviour 
of actors and the problems they are confronted with is applicable knowledge’, 
but this does not necessarily mean that it can be used by the actors. Applicable 
knowledge has generated many prescriptive theories, but these are only rarely put 
into practice by those involved. Knowledge only becomes ‘utilizable’ if it allows 
actors to ask new questions and try out the theories proposed, and if it helps to 
‘enable people to conduct relations based on power, and understand and analyze 
them’ (ibid., p. 305). 

What we call learning about organizations comes at this price: there can be no 
improvement in the way schools function without awareness of the way different 
groups and individuals defend their interests or exercise power in the day-to-day 
operation of a collective body, or of the multiple meanings attributed to their daily 
interactions. There must also be an awareness of the contradictory stimuli emerg- 
ing from these interactions on a daily basis, which often lead to mental blocks, 
resistance to change and repeated deadlock — all of which lead a great many teach- 
ers to conclude that ‘going it alone’ has much more to be said for it than unsuccessful 
efforts to co-ordinate their work with that of their colleagues. 

WHO IS TO BE ACCOUNTABLE? 

In the context of assessment, the importance attached by organizational analysis to 
the school environment strongly challenges the idea that decentralization could 
absolve the central authority or pupils’ families, or indeed local officials, of all respon- 
sibility for schools’ performance. There can be no assessment of a school’s 
performance without an assessment in parallel of the regulatory methods imposed 
by the central authority, which would address questions such as whether the con- 
text in which they operate enables schools to organize themselves independently and 
teachers to be more professional and take responsibility for their decisions. Although 
it is important to assess individual schools, it is also important to assess the actions 
and decisions of the educational authorities, the role played by intermediary offi- 
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cials, teacher trainers and researchers,^^ the strategies pursued by the parents’ asso- 
ciations and the many pressure groups that all think they know what is best for the 
school. In other terms, not everything that goes on inside a school can be dealt with 
inside the school (see Beillerot, 2000). The quality of a school reflects the quality of 
its institutional, social, cultural and economic environment. 

In this sense, there can be no self-assessment without assessment by others. 
One group of actors is probably never able accurately to assess what is its respon- 
sibility and what is the responsibility of another groups of actors (the pupils themselves 
forming one of these other groups). It is therefore only by comparing assessments 
carried out by the different groups of actors that a more accurate view can be formed 
of the responsibility of the various parties involved for the results achieved. This 
will also help the group operating in the school to adjust its own strategies to the 
overall situation. 

Notes 

1 . See B. Favre and F. Osiek, Les ecoles primaires genevoises [Primary schools in Geneva] — 
in press, 

2. See the summary by J.-L. Derouet (1987); see also V. Dupriez (1999). 

3. See, for example, the comments of P. Bressoux (1994) on the concept of the ‘atmos- 
phere’ in a school. 

4. In 1995 the Directorate of Primary Education proposed a reform of the way primary 
schools functioned on the basis of principles such as greater differentiation of teaching, 
leading to a system of cycles rather than having a different grade each year, teamwork, 
and the drawing-up by each school of a projet d'etablissement (a school’s ‘mission state- 
ment’). See: Directorate of Primary Education, Texte d’ orientation [Guidelines], Geneva, 
Cantonal Department of Education, August 1994. 

5. The primary school system in Geneva, for 4- to 12-year-olds, is now divided into two 
four-year cycles, with the division into grades by year being gradually eliminated. 

6. Approximately 40% of pupils in Geneva are of non-Swiss origin. Some schools, main- 
ly in working-class areas, have over 60% of pupils whose mother-tongue is not French. 
See: Service de la recherche en education, 2001. 

7. This principle applies specifically to primary schools, which are located in such a way 
that children can attend a school as close as possible to their home. 

8. For an initial four-year period, seventeen schools volunteered to take part in an explorato- 
ry phase of practical implementation of these general principles. We included three of 
these seventeen schools in our analysis. For some data regarding the assessment of this 
initial phase, see Favre et al. (1999). 

9. See the analysis proposed by H. Amblard, P. Bernoux, G. Herreros and Y.F. Livian 
(1996). 

10. It is likely that in highly decentralized education systems in which parents can choose 
which school to send their children to, these principles of legitimacy define the overall line 
of each school rather than the personal position of each teacher or group of teachers. 

11. Unlike the situation in France, parent associations in Switzerland are not officially rec- 
ognized, although their umbrella association is represented in certain bodies of the 
Department of Public Education; they are set up on a purely voluntary basis; they are 
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seen more frequently and are generally more active in districts and municipalities inhab- 
ited by mainly middle-class and upper-class families (see indicator D5 in Service de la 
recherche en education, 2001). 

12. These were semi-directive interviews, generally lasting an hour-and-a-half or two hours. 

13. Officially, the headteachers cannot be considered as decision-makers; their responsi- 
bilities are essentially of an administrative nature: they provide the Directorate of Primary 
Education, via the inspector, with all information concerning pupils’ attendance and 
the organization of classes. They also have responsibility for certain disciplinary mat- 
ters, e.g. the replacement of absent teachers and organizing the supervision of recreation 
periods. 

14. There was no systematic observation of classes. Classroom teaching was not our main 
area of interest. What we were mainly interested in was to what extent the methods of 
individual teachers were known and communicated to colleagues, and subsequently 
used by all the teachers in a school. 

15. We are referring here to the distinction introduced by Luc Boltanski (1990) between a 
‘system of love’ and a ‘system of justice’, which formed the basis for his theory on the 
subject. 

16. See note 13. 

17. See Favre et al. (1999). 

18. Our observations would tend to confirm the importance of the role played by inter- 
mediary officials in learning organizations, highlighted in particular by Nonaka and 
Takeuchi (1997). 

19. Class taught by two teachers, each working half-time. 

20. See Favre and Montandon (1989). 

21. In the study by Cyril Lemieux (2000) on journalism one finds an illustration, both the- 
oretical and empirical, of the advantages of an approach centred on individual actors 
in a broad sociological context which distances itself from the ‘critical’ sociological 
approach, without presenting a wholesale endorsement of the individual actor’s views. 

22. Each workshop is attended by pupils from different grades, for greater differentiation 
and co-operation between pupils. 

23. As in the position adopted by supporters of methodological individualism. We, for our 
part, refer here to the ‘economics of conventions’ and to the work of Boltanski and 
Thevenot (1991), who do not see actors as simply seeking to maximize their individ- 
ual interests, independently of any reference to values, or to views of what action is fair 
and legitimate in a given situation. 

24. See the assessment carried out in France of strategies frequently adopted by teachers in 
priority education zones (ZEPs). 

25. See Boltanski and Thevenot (1991) and Boltanski and Chiapello (1999). 

26. My italics. 

27. In Geneva, in particular, some researchers played a key role in defining new directions 
for primary schools and in drafting the circulars sent out to schools. 
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In 1989 New Zealand embarked on an ambitious effort to decentralize the gover- 
nance of its State school system. Under a plan known as the Tomorrow’s Schools 
reforms, Parliament abolished the national Department of Education, which had 
overseen State schools for decades, and transferred operational control of the coun- 
try’s nearly 2,700 primary and secondary schools to boards of trustees elected and 
controlled by the parents of current students in each school. The central govern- 
ment continued to fund the schools and to negotiate national teacher contracts 
through a new Ministry of Education, whose mandate revolved around policy rather 
than operational authority. 

Central to the reform was a new system for holding the self-governing schools 
accountable to the public. In this paper we describe how school reformers in New 
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Zealand balanced school autonomy and public accountability and discuss that coun- 
try’s experience during the 1990s. The discussion draws heavily on our book on the 
Tomorrow’s Schools reforms titled When schools compete: a cautionary tale which 
covers the period 1989 through 1998. We have added a brief update on more recent 
policy debates.^ 

The genesis of school decentralization 
in New Zealand 



The groundwork for New Zealand’s move to decentralize school governance was 
laid in 1987 when the ruling Labour Government appointed a Taskforce to Review 
Education Administration, headed by businessman Brian Picot, to suggest a way of 
structuring a decentralized system. The government had already taken similar steps 
in other sectors, notably the national health and welfare systems. Broadly speaking, 
the task force was operating in the context of two sets of forces. We call them the 
democratic-populist and the managerial-business currents." 

THE DEMOCRATIC-POPULIST CURRENT 



The move to self-governing schools as part of the Tomorrow’s Schools reforms was 
consistent with a long-standing tradition of community involvement in schools. New 
Zealanders had created a society with strong egalitarian values that encouraged pop- 
ular involvement in social and political institutions. Every primary school had an 
elected school committee consisting of five to nine local persons that was charged 
with maintaining the school premises, paying the caretaker and organizing local 
fund-raising activities and parent volunteer activities. Secondary schools were run 
by locally elected boards of governors that included both parents and community 
representatives and whose responsibilities extended to the management of finances, 
selection of principals and the hiring of teachers. In neither case, however, did the 
committees or boards have any voice in curriculum, which was centrally controlled, 
or over day-to-day management, which was the responsibility of the principal. 

Despite these formal mechanisms for giving local communities a voice in edu- 
cational policy, pressure began to mount in the early 1970s for even greater parental 
involvement in the running of schools, for giving local schools more operational 
independence from the Department of Education, and for developing closer work- 
ing relationships between schools and their communities. Such themes were sounded 
repeatedly at national education gatherings, and a 1976 report by the department 
had identified relations between schools and their communities as an area that war- 
ranted attention. Such talk, however, produced only modest changes, such as an 
increase in parental representation on secondary school boards. 

Additional support for more community involvement came on behalf of New 
Zealand’s minority community. The mid-1980s brought growing attention to ways 
in which the State education system was failing to meet the needs of significant seg- 
ments of the population, most notably the Maori and Pacific Islanders. It seemed to 
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many of them that the school system, run by a large professional bureaucracy, had 
lost touch with the communities it served. 

The crescendo of calls for greater parental and community involvement in the 
running of schools, was not lost on members of the Picot Task Force. In their report, 
they wrote that the public submissions they had received were marked by ‘a com- 
mon theme of powerlessness — and consumer dissatisfaction and disaffection’, as 
well as by ‘feelings of frustration in the face of a system that too often appeared 
inflexible and unresponsive to consumer demand’. The task force said that alien- 
ation was particularly strong among ‘those who had been failed by the system in 
the past’ and noted that Maori and Pacific Island children are disproportionately 
represented in this failing group. ^ 

THE MANAGERIAL-BUSINESS CURRENT 

Although democratic-populist themes were sounded frequently in the deliberations 
and recommendations of the Picot Task Force, its fundamental mandate was to 
grapple with issues central to a second current that viewed managerial effectiveness 
as the key to school improvement. The committee had been given a businesslike 
name (Task Force to Review Education Administration) and its report bore the 
rather dry title Administering for excellence: effective administration in education. 

Whereas democratic-populists saw decentralization as a way of enhancing the 
community voice in the running of schools, those sympathetic to the managerial- 
business current saw it as a way to improve teaching and learning by locating decisions 
closer to the point of implementation. Task-force members had heard ample testi- 
mony about managerial problems associated with the existing system, including 
numerous horror stories about bureaucrats who favoured process over results. After 
considering a variety of ways to restructure the department, they came to the con- 
clusion that reforming the existing structure was hopeless and that what was needed 
was an entirely new apparatus. ‘Tinkering with the system will not be sufficient to 
achieve the improvements now required’, the task force wrote. ‘In our view the time 
has come for quite radical change, particularly to reduce the number of decision 
points between the central provision of policy, funding, and services and the edu- 
cation delivered by the school or institution’.^ 

Significantly, Picot and his task force did not oppose a strong centrally direct- 
ed State education system. To the contrary, Picot believed that running a system 
that provided quality education to all citizens was an important and fitting respon- 
sibility of the national government. Fie and his committee were simply arguing for 
good management, and one way to do this, they argued, was to devolve as many 
operational decisions as possible to the level where teaching and learning took place.^ 

A TIGHT-LOOSE-TIGHT STRUCTURE 

The cumulative effect of the various currents that went into the Tomorrow’s Schools 
was what may be described as a tight-loose-tight system of school governance. Under 
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such an approach, the goals and missions of the schools would be clear (that is, 
tight), the schools would have significant responsibility for how they operate (the 
loose part), and schools would be held tightly accountable to the centre for out- 
comes. For a number of reasons, New Zealand has not fully achieved a governance 
structure of this type, but its efforts to move in that direction provide insight into 
the challenges of implementing such a structure. 

In opting for a tight-loose- tight system, the Tomorrow’s Schools reforms had 
forced a major rethinking of how New Zealand held its schools accountable. Under 
the system in place before 1989, accountability was built into the process of deliv- 
ering education through a network of regionally based inspectors employed by the 
national Department of Education. Inspectors carried out a variety of operational 
responsibilities, including the hiring and firing of teachers, professional develop- 
ment, allocation of resources, development of new curricula and providing advice 
to administrators. The inspectors were, for the most part, respected educators who, 
in addition to carrying out their formal responsibilities, provided important infor- 
mal services. They offered advice on curricular or personnel matters, and could use 
their contacts to solve thorny problems, such as arranging for the transfer of a teacher 
who was not getting along with his or her principal. Although the nature of their 
jobs was such that the inspectors inevitably provoked anxiety among principals and 
teachers, they were also seen as having an avuncular side, and the harshness of their 
judgements was tempered by the fact that they functioned behind the scenes. 

Such a system was, however, incompatible with the principle of self-gover- 
nance embraced in 1989. Autonomous schools needed to be held accountable, not 
through direct interventions of authority figures but through some sort of quid pro 
quo. Operational freedom came with the understanding that schools would use this 
freedom in the service of the state that created and funded them. In short, account- 
ability was the price of freedom. 

The Education Review Office 

Given the previous system’s roots in the British education system, it came as no sur- 
prise that New Zealand opted for a variation on the inspectorate model as a means 
of assuring accountability under the Tomorrow’s Schools reforms. The initial reform 
legislation established a new agency, the Education Review Office (ERO),^ that 
would be independent of the Ministry of Education — though responsible to the min- 
ister of education — and charged it with providing an independent and arms-length 
evaluation at least every two years of how well each school was performing. The 
ERO sends teams of one to five outsiders, all former teachers, into each school for 
several days. Members visit classes, pour over documents and records and then dis- 
cuss a draft report with the schools. The team then issues a final public report laying 
out both the school’s strengths and its deficiencies. 

In recommending such a set-up, the Picot Task Force in effect abandoned a 
professional model of accountability aimed primarily at helping teachers deal with 
the teaching and learning process in favour of a more management-oriented model 
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that would minimize the possibility of monitors being ‘captured’ by the groups they 
were evaluating. Under the management approach, the focus was on good man- 
agerial practices, and the major purpose was to inform boards about how well school 
staff members were doing, to provide the government with information on the per- 
formance of schools and to provide information to groups outside the system. 

A SHAKY BEGINNING 

The Education Review Office got off to a halting start, in part because the law that 
established it provided only sketchy direction about how the agency should oper- 
ate and what it should do. While the agency’s head, the chief review officer, was 
empowered to send teams into schools to obtain information and to write reports, 
the agency was given no authority to enforce its recommendations. The initial idea 
was that the agency would audit both the financial and the educational programmes 
of the schools (including early childhood centres) and that it would also evaluate 
the quality of the policy advice that the ministry was giving to the minister of edu- 
cation. Early on, however, the financial function was reduced, with full responsibility 
for financial auditing placed with the auditor general, and the agency was stripped 
of its mandate to review the policy advice of the ministry. 

Despite a large operating budget with provision for 306 employees, the new 
Education Review Office initially accomplished little and had trouble determining 
its new direction. One reason for the confusion was the decision by the first head 
to establish ten relatively autonomous regional offices, each with thirty employees 
who in many cases were carry-overs from the old Department of Education. This 
move sent a signal that things might not be too different than in the past. A highly 
critical 1990 review by a committee that was set up to review all of the Tomorrow’s 
Schools reforms urged a major restructuring of the agency and a 40% cut in its bud- 
get. After a year without a permanent head the new agency finally began to develop 
a sense of identity and purpose under Judith Aitken, who was hired as its new chief 
review officer in 1992. Aitken came to the job with some background in teaching, 
but most of her training and experience was in public administration and strategic 
planning. 

TENSIONS OVER THE ROLE OF CHARTERS 

Aitken’s task was complicated by inherent tensions that had manifested themselves 
even before the ink was dry on the Tomorrow’s Schools legislation. The most sig- 
nificant of these tensions related to the nature of local school charters. 

In keeping with democratic-populist principles, the notion was that every school 
would build its programme around a written charter giving formal voice to the aspi- 
rations that local communities held for their schools. Since the schools were part of 
a national system, however, it was also understood from the outset that local school 
charters would be consistent with national curricular and other public interests that 
justify their public funding. 

83 

Prospects, vol. XXXI, no. 4, December 2001 




542 



Edward B. Fiske and Helen F, Ladd 



The fundamental goals and objectives for New Zealand’s education system are 
spelled out in a three-part document titled the national education guidelines (NEGS) 
that emerged in its current form in 1993 and must be incorporated into all school 
charters. These guidelines consist of national education goals and administration 
guidelines, as well as national curriculum statements. The national education goals 
are quite broad and include such ambitious objectives as high achievement for all 
students, equality of educational opportunity and development of the knowledge 
and skills needed by New Zealanders to compete successfully in the ‘modern, ever- 
changing world’. They also call for the advancement of Maori education and respect 
for the diverse ethnic and cultural heritage of New Zealand people. The national 
administration guidelines spell out general responsibilities for the boards of trustees 
in the areas of student achievement, employment and personnel matters, finance and 
property issues and the provision of a safe physical and emotional environment for 
students. 

Charters were intended to be the ‘lynchpin’ of the new structure of compul- 
sory education and, as such, to provide each school with ‘clear and explicit objectives’ 
reflecting ‘both national requirements and local needs’.^ The charters would con- 
stitute a ‘contract between the community and the institution, and the institution 
and the state’. ^ Under this three-way contract the various parties would have dif- 
ferent responsibilities but would relate as equals. The state would fund schools and 
provide national guidelines, while boards of trustees would make local policies and 
run the schools in line with community interests. 

But the language papered over some built-in ambiguities. With primary and 
secondary education compulsory, how much control over missions should the State 
cede to schools? Could the charter ever be a contract in the sense of imposing enforce- 
able responsibilities and obligations on all parties? If so, how would it be enforced? 
Could the government commit itself to provide sufficient funding for a school to 
achieve the objectives in the charter? Working out such details was the task of an 
implementation group, which circulated a draft framework for charters in March 
1989 and a final version in May. These documents addressed the ambiguities by 
substantially altering the design laid out in the earlier documents. 

The March draft made it clear that local school communities (that is, parents 
connected with the school) would have little or no say over about 80% of the con- 
tent of school charters because, to protect the government’s interest in educational 
outcomes, the government was already planning to require that every charter include 
a commitment to the national educational guidelines. The May 1989 framework 
further weakened the power of parents by changing the three-way contract to a 
bilateral agreement between a school’s board of trustees and the Minister of 
Education, leaving no formal governance role for the local community in the form 
of obligatory opportunities for parental input. 

Partly out of concern about the ministry’s insufficient capacity to negotiate and 
approve 2,700 new charters in a short period of time, and partly because the Picot 
Task Force had overestimated the ability of local boards to develop meaningful char- 
ters that were explicit enough to be used as accountability documents, the charter 
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framework was changed once again in January 1990. First, the relationship between 
the two parties was further downgraded from an ‘agreement’ to an ‘undertaking’. 
Second, the new document removed the ministry’s legal obligation to provide ade- 
quate funding to schools, thus turning the former partnership into what amounted 
to a one-way obligation of boards to the State. A subsequent final change, acknowl- 
edging that schools might not have adequate funding to achieve the goals and 
objectives of their charters, altered the language to require only that the schools 
make reasonable efforts to do so within the resources available to them. 

Thus, in a few months the charter went from being a three-way contract or 
partnership to a two-way ‘agreement’ to a one-way ‘undertaking’. As Liz Gordon, 
an academic critic who subsequently became a Member of Parliament as a member 
of the small left-wing Alliance Party, wrote shortly afterwards: ‘The state had taken 
the first step in regaining the power that had been given away in the Picot Report’.^ 
As a result, not only did school communities come to play a smaller role in the char- 
ter development process than had been envisioned but the content of charters became 
homogenized. Under such circumstances charters could not serve as a clear basis for 
accountability. 

The ERO and the schools 

Since the charter failed to emerge as the definitive document envisioned by mem- 
bers of the task force, the Education Review Office faced the task of determining 
for itself what it should be looking at as it carried out its responsibility to monitor 
schools. With a background in public management, Aitken opted for a formal 
approach to review process that involved two distinct types of audit: assurance and 
effectiveness. 

ASSURANCE AUDITS 

The first type of audit, which began in 1992, was designed to assure that the boards 
of trustees were meeting their legal obligations to the Crown as specified in the 
school’s charter, including the National Education Guidelines, and in other agree- 
ments between the State and the schools, such as the school’s property agreement. 
This relatively narrow emphasis on legal compliance meant that assurance audits 
focused on administrative process as opposed to impacts on students or education- 
al outcomes. As Aitken explained to us in an interview, such a focus on compliance 
was essential during the early years of the Tomorrow’s Schools reform when lay 
boards were still learning what was expected of them. During 1992/93, only 12% 
of the boards of trustees were operating in a fully lawful way. By 1998 the propor- 
tion was up to 90%. 

EFFECTIVENESS AUDITS 

In 1993, the ERO added the second type of audits to its arsenal as a means of shift- 
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ing its attention towards student achievement. According to official documents, an 
effectiveness review is ‘an evaluation of student achievement and the impact of the 
teaching services and management practices within a school on that achievement’. 
Such a description is a bit misleading, however, since, in the absence of national 
compulsory tests, the ERO had no reasonable way to compare student achievement 
in one school either with other schools or with national standards. 

Instead, effectiveness reviews were primarily process-oriented and posed two 
specific questions to local boards of trustees: (1) What do you expect the children 
in the school to learn? and (2) How will you know that learning has occurred? Thus 
the ERO sought to encourage schools to focus on student achievement and to imple- 
ment practices, such as better systems of assessing students, that would allow the 
trustees and the principal to gain a better sense of how much students were learn- 
ing. While the reviewers spent time in classrooms and comment in their reports on 
the quality of teaching in schools, they lacked the authority of inspectors in the for- 
mer system to evaluate individual teachers. 

In 1998 the ERO began to consolidate the two earlier forms of revievv into a 
single combined accountability review with the purpose of assisting in improving 
the quality of education for students. The new reviews were supposed to pay less 
attention to the compliance issues that were so central in the early days of the 
Tomorrow’s Schools governance reforms, to give more attention to academic out- 
comes and to put more emphasis on the adequacy of schools’ own self-review reports. 

HOW SCHOOLS RESPONDED TO THE ERO 

One of the first changes that Judith Aitken made when she assumed leadership of 
the ERO was to make all of its reports public and to encourage the local news- 
papers and television stations to publicize them. Given that the ERO had no enforce- 
ment power, she viewed public information as the main policy tool available to her 
to induce the schools to improve. For ambitious principals and teachers, being at a 
school with positive ERO reports had major benefits. In addition, it could have a 
potentially big impact on how attractive the school was to potential students and 
their parents (a factor that became significant with the introduction of parental 
choice in 1991,. 

The site visits by ERO teams and the public reports that they produced elicit- 
ed a wide range of reactions from educators, parents and board members in the 
schools they visited. In a survey of principals of primary schools that went through 
ERO visits in 1995/96, Cathy Wylie of the New Zealand Council for Educational 
Research found that 51% said that the review process had been on balance a posi- 
tive experience, while 49% described it as more negative.'* In practice, most schools, 
especially those without significant enrolment problems, have now more or less 
learned to take ERO visits in stride. Jill Stanley, principal of the Porirua School near 
Wellington, said that since she and her teachers routinely keep rather detailed records 
of student performance, they do not have to spend much time preparing for ERO 
visits. ‘It’s basically a question of tidying up the records we already have’, she said.'^ 
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The Wylie study found that 75% of primary school principals viewed the ERO 
team members as either reasonably or highly professional; yet only 32% felt that 
the review team had sufficient understanding of the particular needs of their school, 
and 24% thought the team had no such understanding. A general view among teach- 
ers and principals seems to be that ERO reports were not particularly helpful on 
important issues relating to teaching and learning. Teachers find the ERO people 
very off-putting’, said Dennis Thompson, principal of Lyall Bay School in Wellington. 
‘Some of the people are not familiar with primary education. They come in the class- 
room not sure of what they are looking for, and some of the statements they make 
are not well founded. There is nothing that the school gets out of the process. We 
have had three glowing reports, and the only recommendation from the most recent 
one told us that the gate on the swimming pool swings the wrong way’.^^ 

Another review of the ERO, this one sponsored by the Post Primary Teachers’ 
Association, the secondary school teachers’ union, also found mixed responses. 
Respondents complained that schools sometimes put their efforts into compliance 
rather than genuine school improvement. They reiterated complaints that review 
teams did not put enough emphasis on what goes on in classrooms and suggested 
that the resulting media reporting generated a negative view of the teaching pro- 
fession. 

Since drawing up the more than two dozen policies required was a time-con- 
suming task for educators and trustees, a cottage industry developed in which 
principals, sometimes for a fee, would fax to colleagues the documents that they 
were obligated to create on particular topics, such as how the school was protect- 
ing animal rights. A major emphasis was also placed on whether teachers were 
keeping adequate records on student achievement. Not surprisingly, principals crit- 
icized the ERO teams for nit-picking and creating unnecessary paperwork. Teresa 
Lilley, the deputy principal of Mt. Albert Primary School in Auckland, recalled that 
one of her teachers was rapped by ERO Tor not having procedures on feeding gold- 
fish’. 

One paradoxical effect of the ERO was to create certain nostalgia for the old 
inspectors. ‘We used to have one attached to the school, and he got personally 
involved in what was going on’, recalled Ashley Blair, principal of Cannon’s Creek 
School in Porirua. ‘He had educational vision and was my mentor. He didn’t just 
carry out a cold and impersonal inspection’.'^ 

The ERO as conscience of the system 

As a review agency that is independent of both the Ministry of Education and schools 
and that reports directly to the minister of education, ERO has no direct impact on 
the policies proposed and implemented by the ministry. Indeed, as noted above. 
Parliament specifically prohibited the agency from offering comments on the qual- 
ity of the advice that the ministry provided to the minister, a function that was part 
of its original mandate under Tomorrow’s Schools. 

Nevertheless, Aitken’s vision for the Educational Review Office extended beyond 
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its role as monitor of individual schools, and she chose to use the power and visi- 
bility of ERO in an indirect manner to challenge national education policies and to 
focus public attention on large structural problems encompassing groups of schools. 
As Aitken put it in a conversation with the authors: ‘It has been a struggle to get the 
Ministry [of Education] to understand that the school is not a great unit to focus 
on. Compare the successful firm. It has a lot of vertical and horizontal linkages to 
other firms’. 

Consistent with her view that the problems facing many of the schools were 
too large to be addressed by the schools alone, Aitken convinced her divided agency 
to publish a series of high-profile reports on particular groups of struggling schools. 
The first and most controversial of these reports emerged in 1996 and focused on 
the forty-five schools in the two South Auckland suburbs of Mangere and Otara, 
most of which served highly disadvantaged and overwhelmingly minority student 
bodies. The report described the region as an educational disaster area character- 
ized by rampant governance and management failure. Specifically, it said that 42% 
of the schools in these areas were performing very poorly or were under-perform- 
ing and that 27% were in the highest category of risk because they had required at 
least one follow-up review by ERO. Trying to be constructive, the ERO couched its 
policy recommendations in ways that it hoped would be acceptable to a ministry 
not inclined towards intervention. For example, it called for the establishment of a 
strategic development centre that would serve as a broker for the management ser- 
vices needed by the South Auckland Schools, exit incentives for poor teachers, and 
recruitment incentives for new, higher quality teachers. 

The report was met with outrage in South Auckland. The local principals, 
including principals at schools that the ERO judged to be relatively successful, were 
distressed at the agency’s apparent insensitivity to the severity of the educational 
challenges they faced, many of which they linked to middle-class flight that was well 
beyond the power of schools to control. The two teachers’ unions and the princi- 
pals’ association were also angered at what they viewed as unfair criticisms of schools 
and teachers. Parents, too, thought the report was unfair on the ground that it inap- 
propriately singled out their schools for criticism. In addition, academics criticized 
the report on grounds similar to those raised by the principals: that it failed to take 
sufficient account of the context in which the schools operate and focused its rec- 
ommendations too much on management issues. 

The report had a major impact on discussions within the ministry and ulti- 
mately prompted the government to establish a US$10 million programme aimed 
at assisting troubled urban schools. Contrary to the hopes of the ERO, however, 
the ministry did not adopt any of its proposals related to teacher recruitment, opt- 
ing instead for a focus on management and community engagement. Subsequent 
reports of this type focused on twenty rural schools in the East Coast and seventy- 
eight schools in Northland — areas that, like South Auckland, have high proportions 
of minority group and low-income students. The East Coast report has encouraged 
the ministry to consider new ways of organizing school resources in that area, includ- 
ing new forms of shared administration. 
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The ERO as promoter of good edycational 
practices 

Under Judith Aitken, the Educational Review Office also carved out a role for itself 
as the provider of information aimed at helping schools do their jobs more effec- 
tively. Using the information it gleaned from its individual school reports, agency 
staff members drew conclusions about what works and what does not work in 
schools and issued periodic reports aimed at principals, teachers and members of 
boards of trustees (Professional leadership in primary schools, Core competencies 
for school principals and The capable teacher). Other such reports were addressed 
to parents (Choosing a school for a five-year old and Choosing a secondary school), 
while still others were focused on general issues related to teaching and learning 
(Addressing barriers to learning and students at risk). All are public documents avail- 
able on the Internet. 

An official review of the ERO in 1997 praised these publications and report- 
ed that they were highly regarded among educators and that they had generated 
useful debate on a variety of issues. An alternative perspective on these reports 
appears in a 1997 analysis of the ERO commissioned by the Post Primary Teachers’ 
Association.'^ The authors criticize the ERO for promoting a ‘good practice’ model 
of schooling that is conservative, rather than a ‘best practice’ model, which they 
argue would be potentially more progressive. They also assert that while the ERO 
claims that its reports are asocial and apolitical, they define the concept of a good 
teacher and a good school through a set of middle-class lenses. 

Policy issues raised 

We have characterized New Zealand’s approach to the balancing of self-governance 
and accountability as a tight-loose-tight system under which the purposes of edu- 
cation are defined by the centre, schools are given considerable latitude in deciding 
how to go about achieving these purposes, and the centre then evaluates the extent 
to which schools succeed in meeting agreed-upon educational goals. Such a system 
makes intuitive sense and is probably the only rational way to reconcile operational 
autonomy and accountability — at least in principle. As we have seen, however. New 
Zealand encountered challenges in connection with each of the three elements. 

TIGHT MISSIONS 

Charter documents were originally intended to articulate a vision for local schools 
that would affirm both national and local goals and, in the process, become a basis 
for accountability. They failed to serve that purpose, however, largely because they 
lacked the necessary specificity. The national goals were broadly defined with no 
measurable objectives and the local goals were typically quite homogenous and, 
again, not specific and measurable. 
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LOOSE OPERATIONAL CONTROL 

Although the Tomorrow’s Schools reforms represented an affirmation of the capac- 
ity of local schools to manage their own affairs, it turned out that a significant 
minority of schools, most visibly those serving low-income students, needed far more 
than operational autonomy and good management to be successful. These schools 
also needed outside assistance in forms ranging from financial support for special 
programmes for at-risk students to professional development of teachers. 

New Zealand’s decision to introduce an arms-length accountability system — 
and thus an explicit policy that ERO would not provide direct assistance to 
schools — became highly problematic. Largely due to the public nature of the ERO 
reporting system, it became evident that many schools in New Zealand were strug- 
gling under the burdens of self-governance. It then became reasonable to ask what 
provisions the government had established to assure that struggling or weak schools 
get the special support they need to carry out their roles as agents in an effective 
manner and to correct the deficiencies identified by the ERO. Strikingly, there was 
little room in New Zealand’s conception of decentralized school governance for the 
buttressing of weak schools. 

In partial recognition of this shortcoming, one review of the Education Review 
Office recommended that, as a normal part of its reporting procedure, the ERO 
include a section on the sources of advice and guidance available in the local area. 
More significantly, it recommended that the Ministry of Education ‘establish a range 
of actions to assist schools where action is required to improve the management and 
delivery of education’. 

By the late 1990s, the non-interventionist stance on the part of the Ministry 
of Education became untenable in the face of mounting evidence that some schools 
were failing for reasons that included forces outside their control. Officials reluc- 
tantly began to develop strategies for intervening in South Auckland and other 
distressed areas. 

One lesson from New Zealand is that both functions — external monitoring 
and school support — are important and that institutions need to be in place to accom- 
plish both of them. While a public agency must assume direct responsibility for the 
monitoring function, support functions could be provided by one or more outside 
agencies, either non-profit or profit-making. The government, however, has a clear 
responsibility to assure that such support is available to all schools. 

ACCOUNTABILITY FOR OUTCOMES 

The Education Review Office has been criticized for its focus on administrative and 
educational processes rather than on student outcomes. To be sure, some of the 
processes are related to outcomes, such as whether the curriculum is being delivered 
and whether boards of trustees have any way of knowing what the students are 
learning. Nonetheless, the reviews often became mechanistic, were heavily focused 
on management procedure:^, and did not necessarily foster better educational out- 
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comes. Moreover, the ERO provided no evidence in support of its general view that 
‘the quality of school governance and management is a reliable indicator of the qual- 
ity of educational services provided’.^’ 

Most people, including Judith Aitken, acknowledge that the absence of com- 
pulsory national tests hindered the accountability process. At the same time. New 
Zealanders have some serious concerns about introducing such tests. They fear that 
they will narrow the curriculum and that they will be misused both by the ERO and 
the public in evaluating schools. The concern about misuse springs from the obser- 
vation that average test scores across schools are highly correlated with the 
socio-economic mix of students in the school. Unless sufficient testing is done to 
permit value-added calculations, or comparisons are only made between schools 
with a comparable mix of students, test results may provide misleading information 
on a school’s effectiveness. 

Even if national test results were available, however, a strong case can still be 
made for the inspectorate model. One reason for this is that the concepts of out- 
comes and school processes may not be as distinct as the tight-loose-tight governance 
structure would require. The public has an interest in assuring good processes as 
well as good outcomes and in strengthening the links between the two. For exam- 
ple, although a healthy and safe school environment is not a measure of educational 
outcomes, it is of considerable importance to the public. In addition, the public has 
an interest in assuring that a school is complying with the terms of its charter and 
with any legal requirements. Is it offering the type of instruction specified in the 
agreement? Is it able to assemble the appropriate staff? Are its interactions with par- 
ents consistent with the agreement? 

Another set of issues relating to the maintenance of ‘tight’ accountability was 
raised by the public nature of the ERO reports on local schools. Are the outcomes 
of such a public process always the appropriate ones? Do the public reports always 
elicit the desired behaviour? And are they fair? 

To the extent that the reports focus on the symptoms of much larger problems 
outside the control of the schools, they may do a disservice to the school’s officials, 
as was argued by the South Auckland principals in response to the report on their 
schools. In addition, the distinction between managerial effectiveness and teaching 
quality can be important. The danger is that a school — and the principal and teach- 
ers within it — get a bad rap in the press because of managerial ineffectiveness even 
though it may be providing a reasonable level of educational services. The principals 
also complained that follow-up reports by ERO on troubled schools tended to stress 
continuing deficiencies while minimizing steps that had been taken to deal with them. 

Two reviews of the ERO came to opposite conclusions about the relative ben- 
efits and costs of the use of public information as a policy lever. One report 
commissioned by the teachers’ union highlighted the concerns of teachers that media 
reporting tends to be fragmentary, negative in tone and ‘crisis-oriented’. It argued 
that public reports on failing schools, many of which serve disadvantaged students, 
are likely to exacerbate the decline of students from such schools, thereby com- 
pounding their problems.^^ 
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The second report, an official review of the ERO, fully endorsed the agency’s 
legal obligation to make all final reports available to the media while suggesting pro- 
cedural changes designed to minimize adverse effects of the sort that had been 
occurring in the past. These changes would include a longer time for schools to 
respond to the initial report so that they would be ready with action plan to address 
the report’s recommendations. The final version would include not only the school’s 
action plan, but also an introduction prepared by the school describing the context 
in which it operated, brief details of significant achievements since the last review 
and issues on which the school was working.^^ 

Looking ahead 

The points made in these reviews and reflected in studies by Wylie and others have 
not gone unnoticed. In December 1999 a new Labour Government assumed power 
in New Zealand, and its Minister of Education, Trevor Mallard, appointed a five- 
member ministerial review committee to take a hard look at the functioning of the 
Educational Review Office. Mallard specifically asked the committee to consider 
whether the State education system would be better served if the ERO were incor- 
porated into the Ministry of Education. 

In its report, titled ‘Ministerial Review of the Education Review Office’ and 
issued in February 2001, the committee concluded that the ERO had made a valu- 
able contribution to the encouragement of compliance and accountability on the 
part of schools and early childhood centres and that it should remain a stand-alone 
agency. At the same time, the report suggested that the ERO and the ministry should 
establish a system for working together on matters of common interest. 

The review committee also took note of complaints that the ERO, while zeal- 
ous in pointing out deficiencies, provided schools with little in the way of constructive 
criticism, especially those serving Maoris and Pacific Islanders. The report called for 
an ‘assess and assist’ model under which ERO reviews would focus not so much on 
the failures of schools but on ways that they might improve. It urged the agency to 
provide follow-up assistance to schools in the form of workshops and visits to mon- 
itor progress. Under such a model, inspectors would pay increased attention to 
self-reviews by the staff of schools they were appraising, and the teams would be 
required to include at least one professional educator chosen by the school being 
reviewed. The net effect of such a model would be to nudge the accountability sys- 
tem back toward the ‘professional’ model that had been explicitly rejected by the 
Picot Task Force. 

How far New Zealand will retreat from the arms-length monitoring model 
remains to be seen. Not everyone agrees that providing advice is an appropriate role 
for the ERO. Were the ERO to become a purveyor of advice as well as criticism, at 
least one pair of observers suggested, ‘Over time, a likely consequence will be for 
ERO to reduce the power, incisiveness and boldness of its recommendations in order 
to increase the likelihood that its advice is seen to be successful’.” They argue instead 
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that advice and assistance should be provided by ‘other agencies \vith specific skills 
and experience’ in particular areas of education. 

Conclusion 

Based on New Zealand’s experience, we endorse in principal the basic model of a 
tight-loose-tight approach to school governance and the inspectorate approach to 
accountability, but we emphasize the difficulties of carrying out that approach in 
practice. Some of the shortcomings with respect to accountability that arose in the 
New Zealand context did so because of the absence of nationally uniform measures 
of student performance. While other countries could readily address that particular 
shortcoming by requiring external testing of students — ideally in a form that per- 
mitted a focus on gains in test scores rather than levels — other challenges would 
remain. 

These challenges arise in part from the fact that educational outcomes are not 
as readily distinguishable from school processes as the pure tight-loose-tight model 
would imply. In order to address that problem within an inspectorate model of the 
New Zealand type, school charters could be required to specify process as well as 
outcome goals. For such charter documents to serve as the basis of accountability, 
however, they would need to include clear objectives related to the various goals 
that were sufficiently specific to be measurable. For goals that are set nationally, the 
additional issue of the adequacy of funding arises, since it would be unfair to hold 
schools accountable for outcomes and processes for which the schools had inade- 
quate resources to achieve. 

As we have described. New Zealand continues to grapple with whether it makes 
sense to separate cleanly the function of monitoring and accountability from the 
function of support and advice. One of the main lessons we draw from New Zealand’s 
experience in this regard is that arms-length monitoring, by itself, will not assure 
that self-governing schools will provide quality education. Quality education will 
also require significant assistance and support for struggling schools. Failure to pro- 
vide that assistance and support in some form could make any arms-length 
accountability system punitive rather than a constructive force for school improve- 
ment. 
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Introduction 

There is a small but important literature (Edwards, Nicoll & Tait, 1999; McClaren, 
1998; Jansen, 2001) on the ways in which particular discourses about teaching and 
learning emerge within education systems as a consequence of the increasing inte- 
gration of such systems under globalization. This literature holds that globalization 
is not simply a process of economic integration on a planetary scale in real time 
(Carnoy, 2000), but also a process of educational integration that is reflected in 
trans-national discourses about educational events such as assessment and exami- 
nations (Edwards &: Usher, 1997), Third World States at the margins of globalization 
are not immune to this process of educational integration (Jansen, 2000). A partic- 
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ularly powerful measure of this process is the concept of performance^ which remains 
a pervasive expression of South Africa’s integration into the economic, political and 
cultural logic of globalization. 

I will begin this article by describing the ways in which the discourse of per- 
formance emerged within the South African context after the legal termination of 
apartheid during the early 1990s, the inauguration of the first democratic and non- 
racial government in April 1994 under the presidency of Nelson Mandela, and the 
installation of a new education system starting in late 1994. The discussion is then 
advanced in an analysis of a particular instance of educational reform in which the 
discourse and practice of performance is prominently expressed, i.e. the new nation- 
al policy on whole-school evaluation. I conclude by drawing the connections among 
the theory, politics and epistemology of performance within the South African edu- 
cation system, and its implications for educational change after apartheid. 

The policy context and the emergence of 
performance assessment 

f 

The history of struggle against apartheid was marked by an emphasis on democra- 
tic participation, on consultative processes, on an ethos of community (rather than 
the individual), and on the value of collective experience (ANC, 1994). The Yellow 
Book, regarded as the base document of the African National Congress’ on educa- 
tion and on which all major government education policies since 1994 have been 
built, claims that: 

We are committed to an open and publicly accountable process of policy development. Indeed 
the democratization of the policy process and of the education and training system as a whole 
lies at the heart of our policy framework. In this respect, the framework builds on processes 
already underway which point to the democratization of the system [...] Participatory, con- 
sensus-building policy development [is something] from which we have learned much (ANC 
1994, p. 5). 



It was surprising, therefore, that the first official policy documents of the new State 
were premised on statements of final outcomes, on expert-driven change, on a ‘top- 
down’ policymaking apparatus that marginalized stakeholder involvement in the 
planning and execution of educational change, and on individual and institutional 
performance (Nxesi, 2001; Jansen, 2001). 

In its general curriculum vision, the State adopted an educational philosophy 
called outcomes-based education (OBE) that permeated all education and training 
policies. As I have shown elsewhere, the precise origins of OBE in South Africa is 
unclear (Jansen, 1999), but its central tenet is that the entire education system should 
be re-engineered away from an emphasis on what the teacher or curriculum ‘cov- 
ers’ in a period of instructional time, towards what learners can actually do (perform 
or demonstrate) as a result of certain educational inputs. 

The school curriculum plan, called Curriculum 2005, listed sixty-six specific 
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outcomes that signal levels of performance that learners were to achieve at various 
stages of their schooling. These specific outcomes derived from twelve legislated 
‘critical cross-field outcomes” that were to guide curriculum planning across all lev- 
els of the education and training system. The ‘norms and standards' for teacher 
education lists more than 140 ‘competences’ that teachers and teacher educators 
should achieve as professionals.'^ The South African Qualifications Authority (SAQA) 
is in the process of implementing a National Qualifications Framework (NQF) that 
lists specific outcomes that must be achieved at different levels of the system, as well 
as ‘associated assessment criteria’ for determining whether individual learner per- 
formance demonstrates that such outcomes have in fact been attained. Accordingly, 
universities and technikons are required to submit all their qualifications in an out- 
comes-based format that specifies levels of performance for each outcome. In short, 
from primary education through higher education, from the first week in grade one 
to the final year of doctoral studies, students are required to perform on the basis 
of officially adopted outcomes. 

The emphasis on performance was by no means limited to the curriculum vision 
of the post-apartheid State for the individual learner. Outcomes have also become 
the standard against which institutional performance is measured. The new National 
Plan on Higher Education (NPHE) introduces a new framework for assessing indi- 
vidual universities and technikons on the basis of institutional performance measured 
against sixteen specified outcomes. To ensure the achievement of these outcomes, 
specific ‘levers’ are identified by government to ‘push’ institutions in the direction 
of these outcomes (South Africa, 2001). Institutions are required, therefore, to spec- 
ify employment targets or outcomes for staff equity in their strategic plans that are 
submitted on a three-year basis to government. Institutions are required to improve 
efficiency against specified targets for annual graduation of undergraduate students. 
Institutional performance in terms of research outputs becomes the new basis for 
calculating further State funding for research. Institutions are also required to spec- 
ify higher levels of postgraduate student enrolments, and to achieve those levels of 
performance or face punitive sanction in terms of State funding. In the short histo- 
ry of higher education in South Africa, such tight control over autonomous institutions 
is unprecedented, even when compared to the obsessive years of State power under 
apartheid. 

The obsession with performance naturally led to a broader public interest in 
measuring and ranking educational institutions. The Sunday Times (South Africa) 
assembled two powerful panels, one to measure the performance of higher educa- 
tion institutions, and the other to rank the top 100 schools in South Africa. The 
higher education assessment had expert support from South Africa’s leading think 
tank on universities and technikons, the Council on Higher Education Transformation 
(CHET). The measures of performance were telling: Higher education institutions 
were ranked, we are told, in terms of efficiency rather than quality, the focus being 
on how well institutional resources were managed. There was a problem, of course, 
in applying a standard measure of efficiency given the vast disparities in resources 
and capacity between white and black universities in South Africa, a direct legacy 
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of apartheid. Data, for example, were not collected in a systematic manner. One 
panellist complained: ‘There are few countries in the world that would accept a sit- 
uation in which a performance study has to rely on 1996 data. The incomplete 
returns for 1997 reflect badly on institutions’ (Ian Bunting in Pretorius, 1998, p. 2; 
emphasis added). The measurement of school performance invested heavily in ‘pass 
rates’ with some moderation applied due to ‘contextual factors’, such as the race 
and socio-economic background of learners. Even with such moderation, white 
schools completely dominated the list of 100 schools given the significance of the 
weighting of this prominent measure of efficiency: pass rates. But the listing of the 
top 100 schools is only one part of a public process of measurement. Worse, the 
government decided to also provide a public listing of the worst schools in each of 
the nine provinces — obviously all black. Such ‘black-listing’ of public schools was 
intended, no doubt, to bring public pressure and scrutiny to bear on these institu- 
tions with the somewhat dubious goal of improving performance. 

Another expression of this obsession with performance is South Africa’s enthu- 
siastic participation in international and comparative studies of achievement. South 
Africa is one of the few African countries involved in the TIMSS (Third International 
Mathematics and Science Study) and TIMSS-Repeat studies that measure grade eight 
learner performance in science and mathematics across thirty-eight countries. South 
Africa also participates in the so-called ‘monitoring of learning achievement’ stud- 
ies of UNESCO that compares literacy and numeracy levels among grade four 
learners. The results of these studies make for intense and extended discussion in 
the media (alongside ratings on the world competitiveness scale) about South Africa’s 
relative position with respect to learner performance. 

In a short period of time, therefore, an 2 Lnti-apartheid politics premised on the 
primacy of process had been displaced by a post-apartheid politics premised on the 
primacy of performance. Such performance was to be measured in terms of discrete 
outcomes using specified assessment criteria. This orientation in the post- apartheid 
education and training system had pervaded every major policy position of the new 
State. I now turn to a detailed analysis of the politics of performance in South African 
education using the new policy on whole-school evaluation as a case in point. 

The national policy on whole-school evaluation 

In June 2000, the new Minister of Education (Professor Kader Asmal)'^ introduced 
the National Policy on Whole-School Evaluation (NPWSE) as follows: 

This national policy on whole-school evaluation introduces an effective monitoring and eval- 
uation process that is vital to the improvement of quality and standards of performance in 
schools [...] The findings must be used to re-orientate efforts towards improving the quality 
and standards of individual and collective performance (South Africa. Department of Education, 
2000, p. 7). 

It was immediately clear from this policy that institutional (the school) and profes- 
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sional (the teacher) performance would, for the first time, be brought into sharp and 
systematic focus under the new South African Government. The key proposals in 
the policy are the following: 

• That schools will initiate a process of self-evaluation during which each school 
‘provide[s] an account of their current performance’ (Department of Education 
2000, p. 11). In this process, 

• ‘All members of a school should take responsibility for the quality of their own 
performance. Whole-school evaluation seeks to measure the contribution of 
both staff and pupils to the school’s and their own performance’ (p. 12). 

® That schools are then subject to a pre-evaluation survey by an accredited super- 
visor ‘to build a brief profile about the general level of functionality of the 
school’ (p. 15). During this visit the self-evaluation reports, along with other 
school records, are used to generate the school profile. 

• That schools are then reviewed and assessed through an external evaluation 
process by four to six accredited supervisors over a period of three to four 
weekdays targeting nine focus areas (such as learner achievement) with fol- 
low-up surveys (if necessary) every six to nine months in the context of three-year 
evaluation cycles. 

• That schools are supported through a post- evaluation process by district sup- 
port teams ‘to implement the recommendations of the evaluation report through 
school improvement planning that sets clear targets, priorities, time frames and 
resource allocation’ (p. 16). 

The key areas for evaluation are the following: 

• Basic functionality of the school. 

• Leadership, management and communication. 

• Governance and relationships. 

• Quality of teaching and educator development. 

• Curriculum provision and resources. 

• Learner achievement. 

• School safety, security and discipline. 

• School infrastructure. 

• Parents and community. 

The evaluation will be based on three types of indicators, namely, input indicators 
(such as learner characteristics, funding levels and number of staff), process indica- 
tors (such as quality of teaching) and output indicators (such as standards of 
achievement and attendance rates). There are discrete ‘performance ratings’ that 
will be used on a scale of 1-5 with ‘1’ signifying an ‘unacceptable’ rating and ‘5’ an 
‘outstanding’ score. ' 

The policy on whole-school evaluation, as described above, is important because 
of the ways in which its frames the discourse of performance in South African schools. 
In this regard, there are several tensions that are generated in the policy proposals. 

First, there is tension between school autonomy and state control. At a first 
glance, it appears that schools are being granted greater autonomy to decide on their 
own progress, plans and priorities for school improvement.^ After all, it is the school 
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that measures itself through the self-evaluation process, and these internal docu- 
ments form the basis for subsequent external evaluations by departmental supervisors. 
Indeed, ‘the authority for the professional management issues of the schools will be 
vested with the principal of the school, supported by the professional staff’ 
(Department of Education, 2000, p. 20). This means that the principal and staff 
play a key role in the evaluation processes and in the production of a school improve- 
ment plan. Moreover, ‘all evaluation activities must be characterised by openness 
and collaboration. The criteria to be used in evaluating schools, therefore, must be 
made public’ (Department of Education, 2000, p. 12). 

However, on closer observation, there are critical areas in which the school 
principal and staff are excluded from the evaluation process. Consider the follow- 
ing exemption regarding principals: 

S/he will participate in the evaluation process by attending meetings, interpreting evidence 
and clarifying uncertainties but will not be part of decision-making when judgments about 
the school’s performance are made (Department of Education, 2000, p. 21). 

In addition, while the co-operative aspects of the evaluation are listed throughout 
the key policy document, the external evaluators have legal authority to enter and 
act on a school. Accordingly, 

Through the legal responsibilities bestowed on the minister of education, accredited super- 
visors have the right to enter any school and carry out an evaluation (Department of Education, 
2000, p. 13). 

Furthermore, while schools may lay a complaint about ‘unfair treatment or unjus- 
tified action’, the minister of education remains ‘the final arbiter in any complaint’s 
procedure’ (Department of Education, 2000, p. 14).^ 

There is a second tension in the whole-school evaluation between development 
and accountability. On the one hand, the policy emphasizes the positive benefits for 
school improvement that comes through internal and external evaluation. Schools 
will receive district support and development assistance to implement their improve- 
ment plans. Schools with benefit from a budget provided to the district to assist 
implementation. Schools will be exposed to well-trained district officials who will 
monitor and evaluate the performance of each school. In the words of the minister: 

Whole-school evaluation is meant to be supportive and developmental rather than punitive 
and judgmental. It will not be used as a coercive measure, but will ensure that policies are 
complied with. It will also facilitate support and improvement of school performance using 
approaches of partnerships, collaboration, mentoring and guidance (Department of Education, 
2000, p. 8). 

But what happens if a school does not attain the levels of performance articulated 
in school improvement plans.^ Here the policy faces a credibility crisis among prac- 
titioners. The largest teacher’s union in the country, the South African Democratic 
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Teachers Union (SADTU), has dismissed the policy as nothing more than the old 
‘inspection system’ used under apartheid to force schools into compliance with the 
State’s philosophy and curriculum. The ‘inspection system’ remains a powerful lens 
through which the unions interpret this new policy, even though the minister promis- 
es that it is ‘less punitive’ (Department of Education, 2000, p. 7). 

Since new policies are interpreted in relation to other existing policies, there is 
the added problem that the matriculation examination results are in fact used to 
blacklist under-performing schools in a very public and humiliating process. Indeed, 
the minister’s report on examination results places considerable emphasis on the 
number of under-performing schools in the system — which are listed by name and 
with all their results/ It is difficult, therefore, to convince practitioners that the pol- 
icy on whole-school evaluation will not result in some form of reprimand or exposure 
given how other policies are being implemented in South Africa. The exclusion of 
school staff from final judgements on their performance does not help to build a 
developmental understanding of the new policy. 

At the end of the day, this new policy, in its own words, is about using per- 
formance as a measure of compliance and accountability of the school system to 
national policy. The policy aims: 

® To increase the level of accountability within the system (Department of 
Education, 2000, p. 11). 

® To show to what extent [schools] satisfy the expectations of government and 
the public and how well they are responding to their accountability for the out- 
comes of schooling (p. 11). 

• To ensure that policies are complied with (p. 8, 9). 

There are political consequences if a lack of compliance with national policy and 
the constitution is observed; thus, ‘should the evaluation reveal problems in com- 
plying with the provisions of the constitution, the political head of education in the 
affected province would have to account to the minister in writing within ninety 
days’ (Department of Education, 2000, p. 9). 

The meaning and implications of 
a performance-driven pedagogy 

As noted earlier, the rise of performance-based pedagogies is part and parcel of a 
broader set of globalization discourses that have come to characterize educational 
reforms in South Africa and elsewhere. Indeed, a casual search of the Internet will 
yield 746,000 Web page matches on ‘performance-based education.’ The trans- 
national dissemination and adoption of discourses about flexibility, performance, 
terminal outcomes, quality assurance, efficiency, lifelong learning, standards-based 
assessment, competency, modular education, and national qualifications frameworks 
are not coincidental. Rather, globalization discourses about this trans-cultural ped- 
agogy are made possible through the incorporation of economic (and, to some extent, 
political) systems under contracted conditions of time and space made possible by 
new technologies.® South Africa, as I have shown elsewhere, has inserted itself self- 
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consciously within this pervasive discourse about globalization and associated ped- 
agogies (Jansen, 2000, p. 6), even though its realization in classroom practice has 
yet to be observed. 

But for a country on the margins of globalization, operating under typical Third 
World conditions and with a recent history of racial oppression and inequality, what 
are the implications of performance assessment (or performativity, as Lyotard [1984] 
called it) for educational change? 

First, the national obsession with educational outputs diverts attention from 
the educational inputs required to redress the historical inequalities that continue 
to bedevil the education system after apartheid. Citing the size of the teacher salary 
budget, in excess of 80% of the national budget for education, government acknowl- 
edges the lack of resources for learning materials, teacher training, school 
infrastructure and upgrading, and curriculum development. For all the policy hype 
about performance and delivery as an end product, there has not been a propor- 
tional investment in educational inputs. A very good example is the national policy 
on whole-school evaluation. There simply is no way in which government will be 
able to generate the kind of budget that can provide four to six trained and accred- 
ited supervisors to 29,000 schools every three years for three to four weekdays per 
year to cover nine focus areas in multiple school subjects. Yet, if the policy is to be 
believed, schools and their inhabitants are required to nevertheless perform accord- 
ing to clearly defined outcomes. 

Second, the national obsession with standardized performance threatens to 
undermine fundamental commitments to equity. What a standardized measure of 
performance means is that all teachers or learners or schools, irrespective of their 
history, resources or capacity, are required to attain the same levels of achievement. 
In another context, writes Blake, 'performativity obscures differences, requiring 
everything to be commensurable with everything else, so that things can be ranked 
on the same scale and everyone can be "accountable” against the same standards. 
This in turn entails the devaluing, and perhaps the eradication of what cannot be 
ranked’ (Blake et al., 1998). In South Africa, a white urban school with middle-class 
parents, an established school infrastructure, and an elite group of advantaged stu- 
dents, is measured on exactly the same basis as a black rural school serving the 
children of poor families in dilapidated buildings where a poor ‘culture of teaching 
and learning’ exists. Worse, as recent studies have shown, new policies in fact increase 
the distance between rich and poor schools because of the capacity of the former to 
manage, interpret and implement the policy in its favour. Again, whole-school eval- 
uation will be implemented with well-trained teachers (and possibly external 
consultants) who would provide detailed school improvement plans that will be sys- 
tematically implemented to ensure progress beyond already high levels of achievement. 
Most schools, black and poor, are unlikely to contain or receive such expert levels 
of support from the government. In short, the equity gap will in fact increase because 
o/* policy, and the end measures of performance will remain silent on variables such 
as ‘opportunity to learn’ or constructs such as ‘contextual validity’ (Jansen & Christie, 
1999). 
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Third, the national obsession with performance based on measurable outcomes 
tends to overlook the many different ways in which schools, teachers and learners 
can ‘perform’. This fact may explain the tendency of South African policymakers 
to over-specify outcome statements for the sake of completion. There are sixty-six 
learning outcomes in the new curriculum and the 140 ‘competences’ for teachers 
are vivid illustrations of this problem. The problem, of course, is that once specifi- 
cation begins, there is an infinite list of outcomes or behaviours that can reasonably 
be stated for individuals or institutions. The result, inevitably, is a long list of behav- 
ioural outcomes that is unmanageable, dense and bureaucratic — and, therefore, 
difficult, if not impossible, to implement. 

Fourth, the national obsession with performance privileges external behav- 
iours that are easily ‘codified’ into discrete outcomes. In the words of the most 
prominent advocate of outcomes-based education: 

Outcomes are CLEAR, OBSERVABLE DEMONSTRATIONS of student learning that occur 
at or after the end of a significant set of learning experiences. They are NOT values, attitudes, 
feelings, beliefs, activities, assignments, goals, scores, marks, or averages, as many people 
believe (Spady, 1998, p. 3, emphasis in the original). 

Yet, not everything worth doing in schools can be measured in a set of discrete out- 
comes. Schools, therefore, that build strong cultures of anti-racism among learners, 
or who foster democratic participation in the community, or who build co-opera- 
tive cultures among teachers, or demonstrate high levels of curriculum innovation, 
are not taken as seriously in the performance as than those who can demonstrate 
high levels of achievement on measurable outcomes. Despite the acknowledgment 
of input and process variables, the demands for compliance and accountability in 
an education system that privileges high-stakes examinations at the end of school- 
ing, almost certainly means that undisputed measures of ‘output variables’ will be 
more highly valued under whole-school evaluation. 

Fifth, the emphasis on measured performance in an outcomes-based system 
leads to and encourages the fragmentation of knowledge into ‘bits and pieces’ of 
manageable ‘things’., Reflecting on the experience of one of many standards gener- 
ation bodies (SGBs) in South Africa, Wally Morrow recalls: 

By far the most fundamental difficulties have been epistemological — namely the conception 
of knowledge, which, not clearly articulated, weaves through the discourse(s) [...] At the root 
of this epistemology lies an atomised conception of knowledge (‘bits’ of knowledge), the 
assumption that all knowledge has the same logical form, and a conviction that all language 
is ‘transparent’ and merely a tool for naming an independent reality (Morrow, 2001, p. 11). 

Sixth, the emphasis on performance at the end of a process deflects attention from 
the meaning and value of the ends themselves. Not once in the South African 
policymaking experience has there been a broad and sustained debate on ‘what is 
worth learning’ in schools and universities. Rather, having defined the ‘ends’ or ‘out- 
comes’, the task that remained was to find the most efficient ways of becoming good 
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at achieving those ends or demonstrating the outcomes. The point is not simply a 
procedural one; it is a political concern, for at the root of all curriculum planning 
is the deeper question: ‘What knowledge is of the most worth’? On the other hand, 
‘Under performativity, deliberation over ends is eclipsed [....] All kinds of business 
and activity are measured and ranked against each other, with ever less concern for 
the rationale for doing so’ (Blake et al., 1998). 

Conclusion 

The South African obsession with performance-based pedagogies, as I have shown, 
has negative implications for resolving equity problems in educational reforms; it 
threatens to negate a political debate about ‘goals’ in favour of a technician’s debate 
about ‘ends’; and it fragments knowledge into meaningless tasks that assign value 
to external behaviours rather than the multiplicity of ways in which learning and 
valuing can be experienced (if not always expressed). The real danger to building a 
strong democratic culture through education is that what should be vibrant debates 
about ‘what’s worth knowing’ could be effectively silenced in a performance assess- 
ment system that only values, through a complex assessment system, that which is 
worth doing. Such an understanding of education is, unfortunately, entrenched in 
a global network of economic and technological processes that make such pursuits 
appear both normal and inevitable. 

Notes 

1. The African National Congress (ANC) is the former exiled liberation movement and 
now dominant political party in South Africa, of which Nelson Mandela was president. 
In both posl-apartheid elections (1994 and 1999), the ANC won close to two-thirds of 
the popular vote, giving it a dominant political role inside and outside government. 
Despite its more radical tradition from its days in exile, the ANC as well as the ANC- 
dominated government is now regarded as having much more ‘moderate’ economic 
policies — a result of many factors, including political settlement in 1994 with the white 
minority government and economic settlement with powerful international forces that 
moderate the South African State and economy after the collapse of communism. 

2. The twelve ‘critical cross-field outcomes’ are generic statements of performance that 
hold across levels of learning (different school grades) and kinds of educational pro- 
grammes (science or languages). Among these generic outcomes are critical, problem 
solving skills and learning to work in teams. 

3. The regulated performer in the so-called ‘norms and standards’ has seven well-described 
roles, each broken down into three broad ‘applied competences’ (practical, founda- 
tional and reflexive), each of these broken down into about 140 ‘discrete competences’. 
For example, the role described as community, citizenship and pastoral role requires 
that ‘the educator will practise and promote a critical, committed and ethical attitude 
towards developing a sense of respect and responsibility towards others. The educator 
will uphold the constitution and promote democratic values and practices in schools 
and society. Within the school, the educator will demonstrate an ability to develop a 
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supportive and empowering environment for the learner and respond to the educational 
and other needs of learners and fellow educators’. 

4. The first post-apartheid minister of education under the Mandela Administration was 
Professor Sibusiso Bengu who held the post from 1994-99. Professor Kader Asmal 
became the second minister of education since 1999. Both ministers are members of 
the dominant African National Congress. 

5. Traditionally, South Africa’s public schools have had very little autonomy in deciding 
on curriculum or staffing or school finance and organization. These common areas of 
school operations were tightly controlled by the apartheid government through direct 
political surveillance in schools, as well as through a host of legislative arrangements. 
After 1994, when the new government of Nelson Mandela was installed, schools had 
much greater freedom in deciding on curriculum, pedagogy and even as to how many 
additional teachers could be appointed using school -generated revenue. In practice, the 
former white schools, where the class base of the parents was generally quite high, were 
able to use these democratic spaces to a much greater degree than their black counter- 
parts. 

6. This discussion on the tension between autonomy and accountability will play itself 
out very differently in practice, if recent experience with curriculum change is a guide. 
That is, while formal policy stipulates certain behaviour from practitioners, the lack of 
resources for school-by-school surveillance often leads to schools ‘doing their own thing’ 
with respect to implementation, including sustaining the status quo. Further research 
is required on the de facto expression of formal policies in post-apartheid classrooms. 

7. The school test results are published first in an official government listing, which are 
then often published in part by the national and regional media, often as part of a gen- 
eral report on extreme performances (the best and worst performing schools). 

8. In this respect, it is useful to distinguish ‘internationalization’ from ‘globalization’. The 
former represents a long historical process in which States have imitated each other in 
adopting, for example, mass education or subject-based curricula. The latter represents 
a more recent and complex process in which new educational forms represent an expres- 
sion of the contracted economic conditions in which States function. 
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SCHOOL AUTONOMY AND EVAlUATrON 

SCHOOL EVALUATION 
IN CHILE: 



THE CASE OF SNED 

]uan Casassus^ 



One of the criticisms levelled at educational reform^ has been that significant increas- 
es in national education budgets have not been reflected in any notable improvement 
in pupil achievement. In a review of recent literature on the subject, Mizala and 
Romaguera (2000) argue that what is needed is a better system of incentives for 
pupils, teachers and management in state schools. Nevertheless, the debate on the 
advantages and disadvantages of incentive systems has not been conclusive. Criticism 
has focused on merit pay,^ noting that individual merit pay has two important short- 
comings. One is that it is very difficult to make a clear association between pupil 
achievement and a specific teacher. The other is that the system encourages oppor- 
tunistic individualism in teachers and generates a competitive style that has an adverse 
effect on the climate in schools, a factor that is considered central to improving the 
quality of education (UNESCO, 2000). In Chile, an incentive system has been devel- 
oped that works differently. The School Evaluation System does not focus on 
individual teachers, but rather rewards management effectiveness in a school and 
the work of the teaching staff as a whole. 

The Chilean education system has a unique combination on the one hand of 
mechanisms encouraging competition among schools and on the other of central 
government support. School administration has been decentralized to municipal and 
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private institutions known as ‘providers', who are responsible to the State for keep- 
ing the school running. The system thus consists of subsidized non-fee-paying schools 
(municipal and private), private fee-paying schools and the bodies that administer 
technical and vocational secondary schools. It is managed by the Ministry of 
Education, both centrally (in Santiago) and in the country's thirteen administrative 
regions. Various kinds of information systems have been set up to assist in the man- 
agement process, including measuring and evaluation systems which are primarily 
intended to improve decision-making. 

The measuring and evaluation system which has evolved over time has four 
components: (i) the Academic Aptitude Test, created in 1967, which regulates entry 
into higher education, establishing the prerequisite of having completed secondary 
education; (ii) the System for Information on and Evaluation of Education Quality 
(SIMCE), which began life in 1982 under the name of PER. This system measures 
pupil’s performance in the fourth and sixth grades of primary education and the 
second grade of secondary education; (iii) the Subsidized Schools Performance 
Evaluation System (SNED), created in 1995; and (iv) a system for evaluating edu- 
cation professionals individually, which is connected to the academic achievements 
of teachers (currently under discussion). The present article concentrates on SNED. 

Education policy In Chile 

The management of the education system is based on education policy. A panoram- 
ic view of the education policies that have been applied and have shaped the system 
can thus be expected to shed light on the present system of administration. 

Education in Chile has long been an object of unremitting attention. For more 
than forty years, the education system has been constantly under observation and 
reform. We shall concentrate on the policies of the military regime (1973-89) and 
those of the democratic regime (1990-2001) that followed. The former concentrat- 
ed on introducing a new administrative framework that weakened the role of the 
Ministry of Education. The latter were designed to improve the quality and equal- 
ity of education, strengthening the Ministry of Education's capacity for intervention. 
There was continuity between the two regarding the general direction of the sys- 
tem's administration, but the focus of education policy, relations with civil society 
and the role of the ministry took a new turn. 

Under the military regime, in a context of authoritarianism and rigid social 
control, educational reform was aimed at institutional change, in particular changes 
in the overall management framework."* The policies of the 1980s focused on reduc- 
ing the sphere of influence of the Ministry of Education, and privatization and market 
forces were encouraged as a means of regulating and giving new impetus to educa- 
tion. However, this approach was adopted in the context of a bureaucratic, 
authoritarian State with markedly centralizing tendencies and supervised by the 
Ministry of the Interior,^ which in effect put a damper on market forces. 

When a democratic regime was established in 1990, it was decided not to 
change the system's administrative framework. Nevertheless, shortcomings with 
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Table 1. Comparison of educational policy under the military and democratic regimes. 



Educational policy Educational policy 

under the military regime under the democratic regime 

1973-89 1990-2001 

Decentralization. Public administration was Decentralization. State schools continued to 
reformed in 1980 with the introduction of be administered by the municipality, but 

the concept of the subsidiary State. efforts were made to give them more auton- 

Responsibility for school administration was omy as regards the curriculum and fund- 

devolved from central government to munic- raising. As mayors are now elected by 

ipal authorities. In practice, the legal the general public, it is considered that 

framework was not one of decentralizing, but municipal authorities are decentralized, 
of ‘deconcentrating’ the education system to ‘Deconcentration’ applies to the provincial 
municipal authorities, where the chief deci- and regional levels, 
sion-makers were mayors appointed by the 
central government.^ This was referred to as 
the ‘municipalization’ of education. Sixty per 
cent of schools in Chile are municipal schools. 

Market approach to education. The driving Market approach to education. The system 

force of administrative reform was the intro- of subsidized private schools has been main- 

duction of a market approach to education, tained, as has the spirit of competition among 

with a view to stimulating competition among schools. Nevertheless, efforts are being made 

schools, and the provision of private educa- to strengthen the role of the state through 

tion was encouraged by the allocation of compensatory programmes in the context of 

public funds to non-fee-paying private a policy of positive discrimination in favour 

schools. This led to the emergence of subsi- of schools with low achievement levels in 

dized private schools, under the responsibility under-resourced areas. The education system 

of a State ‘provider’. They account for 30% is thus no longer regulated by market forces 

of schools. alone. 

Subsidies based on demand. This policy was Subsidies based on demand. The USE fund- 

part of the privatization process by which the ing system has been maintained. What has 

traditional form of funding, aligned on expen- changed has been the priority given by the 

diture in the past, was replaced by a government to the education sector, reflect- 

mechanism known as the Education Subsidy ed in a substantial increase in its budget. The 

Unit (USE) whereby financial resources were increase affected the value of the USE, which 

allocated on the basis of pupil numbers,^ the went up from $8,000 in 1988 to $12,000 in 

amount of USEs received by each school 1996.® 

being determined on the basis of the average 
monthly pupil attendance. As a result, schools 
began to compete for pupils and made every 
effort to keep them in the classroom. 

Teachers' contracts. The ‘municipalization’ Teachers' contracts. In 1991 a statute for 

of education meant that teachers ceased to teachers was approved, according to which 

be civil servants and lost their status as teach- central government intervenes in decisions 










568 



Juan Casassus 



ers employed by the State. They became 
municipal employees, and the terms and con- 
ditions of their employment were negotiated 
as in the private sector. 

i 


affecting teaching staff, such as decisions on 
wages (setting a minimum wage), contracts 
and incentives (bonuses for experience, work 
in underprivileged areas, further training, and 
special responsibility). 


Role of the State. The central ministry made 
institutional changes whereby it no longer 
carried out the functions of production, 
administration, provision and direct control, 
which were transferred to inspectors. 
Administrative responsibilities were trans- 
ferred to the Ministry of the Interior, at the 
same time as competence in specific areas was 
‘deconcentrated’ down to the municipal 
authorities. Privatization of the system was 
encouraged. 


Role of the State. On the basis of the previ- 
ous administrative structures, a proactive and 
more vigorous role is played by the State. The 
capacity of the centre, particularly its capac- 
ity to make policy and design curricula, 
intervene in schools, and conduct monitor- 
ing and evaluation, has been strengthened. 
The result is a two-way process in which 
greater decentralization is accompanied by 
greater centralization. 


Management of the system. Management 
style was authoritarian and self-referential. 
Decentralization transferred responsibilities 
to the private sector and cut back the man- 
agement functions of the central ministry. 
The policy focused on expanding the system, 
the overall regulation of which was conducted 
by prescriptive means, by resource allocation 
and by input management. 


Management of the system. Management 
became more democratic and open to civil 
society, encouraging initiatives from stake- 
holders and operating by consensus, at the 
micro-level within the education community 
and at the macro-level with the demands of 
the new actors of civil society (production 
sectors, teachers’ union, churches). The focus 
of the policy shifted towards quality and 
equality as regards the distribution of acad- 
emic performance, so regulation was 
conducted by prescriptive means and by 
process and performance management, which 
required information and evaluation systems, 
and worked on the basis of incentives. 



regard to the quality of the education provided and the evenness of its distribution 
made it necessary to increase the education budget and formulate compensatory 
policies. The policy of democratic governments has been aimed at improving the 
quality and equality of education by increasing the (meagre) education budget, 
improving the conditions of employment of teaching staff, focusing attention on 
under-resourced schools with poor academic performance, transforming curricu- 
lum content and improving quality in the system as a whole. 

In short, education policy over the past thirty years in Chile has been concerned 
with improving the system mainly by generating changes in the overall framework 
of education. The assumption has been that if changes are introduced in manage- 
ment and the management environment, changes in processes and performance will 
follow. The new approach to management has involved: (i) introducing market 
mechanisms by allowing schools more room for manoeuvre and awarding subsidies 
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on the basis of demand as a way of stimulating competition between schools; (li) mak- 
ing management more flexible by opening schools up to the community and enabling 
greater participation by parents; and (iii) focusing the system’s operation on the indi- 
vidual school. These institutional changes called for a mechanism producing 
information making it possible to monitor a system in which a large measure of 
management autonomy had been introduced. This led to the development of infor- 
mation, measuring and evaluation systems. 

SNED 

Decentralization creates the problem of how to manage a system in which, although 
schools are no longer the direct responsibility of the Ministry of Education, society 
as a whole persists in the belief that the central ministry is responsible for education 
and can therefore be held to account for it. A process of decentralization, involving 
competition between schools to improve the quality of the service and attract pupils 
thus co-exists with the centralizing rationale of education management. The mea- 
suring systems that have been developed in this context are seen as instruments for 
performance management; and it is on the basis of the information they provide 
that the central ministry allocates additional resources. 

SNED was designed as part of a performance management policy within which 
it constitutes a framework for evaluating the individual performance of schools in 
a context of both decentralization and centralization. It is administered from the 
central ministry and its objective is to create a system of financial incentives and 
recognition for staff in the best-performing primary schools. 

SNED was set up in 1995 by means of Law No. 19.410 to reward professional 
activity that helps to achieve the objectives of education policy. The law introduced 
a system of economic incentives for schools whose performance was considered to 
be outstanding. Known as Subsidies for Outstanding Performance, they take the 
form of a monthly sum per pupil. The teachers receive a bonus worth approximately 
$500, which is the same as an average monthly wage. The schools receiving this 
award must not account for more than 25% of the regional school roll. This means 
that, overall, these schools represent 25% of the country’s subsidized schools, but 
no more than 25% of the total in any one of the administrative regions into which 
Chile is divided. Of the amount received, 90% must go to the educational profes- 
sionals working in the selected schools, whilst the distribution of the remaining 10% 
is decided locally. The selection is made once every two years. 

The system is administered by the Planning and Budget Division of the Ministry 
of Education. However, the analysis and evaluation are carried out by external con- 
tractors. To date, they have been carried out by the University of Chile. 

Factors 

The law introduced a mechanism to measure performance together with the intro- 
duction of the award. It focuses on performance levels in the following six areas: 
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o Effectiveness: measured by pupils’ academic performance; 
o Improvement: achievement differentials in academic performance of pupils 
attending the same school; 

o ' Initiative: the capacity of the school to incorporate educational innovations 
and obtain the support of external agents in educational work; 

® Improvement of working conditions of teachers and satisfactory functioning 
of the school; 

o Equality of opportunity: arrangements enabling pupils to have access to the 
school and to maintain their attendance, in addition to the inclusion of pupils 
with learning difficulties; 

® Integration and participation of teachers, parents and guardians in the school’s 
educational programme. 

A characteristic feature of SNED is its composite nature. It rates schools’ ‘perfor- 
mance’ in terms not only of their pupils’ performance and progress, but also using 
such important factors for the improvement of educational quality as initiative, 
teachers’ working conditions and participation of the community. It is significant 
that academic results are not taken as the sole measure of a school’s success. Clearly, 
not all the possible areas of achievement of a school have been taken into account, 
but at least a more satisfactory way of representing the results of the efforts of the 
educational community is being sought than simply taking academic performance 
as the yardstick. 

Another noteworthy aspect is that SNED does not consider schools in isola- 
tion but in their environment. It is obviously more difficult to obtain good results 
in situations of deprivation and vulnerability than in favourable social, cultural and 
economic conditions. This is a significant point, since the measurements made by 
SIMCE only provide information on academic performance. Of course, SIMCE’s 
results do specify socio-economic levels and status (fee-paying private, subsidized 
private and subsidized State schools), but only for the purposes of classification. 
This information is not considered as a factor enabling results to be weighted. For 
this reason there is no point in comparing one school’s performance with another 
on the basis of SIMCE results, since there is no mechanism for adjusting for the fac- 
tors that affect their performance. 

Forming homogenous groups of schools 



An effort has been made in SNED to ensure that schools’ performances can be com- 
pared. This has involved forming ‘homogenous’ groups of schools. They are grouped 
not only by region, as mentioned above, but also on the basis of external factors 
that might affect their performance. In fact three types of variables are used: (i) geo- 
graphic variables: rural or urban area; (ii) variables within the system: level of 
education (primary or secondary); and (iii) social and economic variables (house- 
hold income, family expenditure on education, educational level of parents and the 
vulnerability indicator established by the National School Assistance and Grant 
Board (JUNAEB)). Schools can then be grouped according to the social and eco- 
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nomic levels of pupils’ families. Four groups are formed on the basis of the first two 
variables (urban primary schools, rural primary schools, urban secondary schools 
and rural secondary schools). Socio-economic variables are then applied to form 
groups by means of cluster analysis. For SNED’s 2000-2001 survey, 104 homoge- 
nous groups were formed in the country. 

Once the schools have been divided into homogenous groups, SNED measures 
the difference between schools in the same group, with the aim of identifying vari- 
ations in performance. SIMCE answers the question: ‘How is my son or daughter 
performing?’, but does not answer such questions as: ‘How is my son’s or daugh- 
ter’s class, or school, performing?’ (Thomas, 1998). When the focus is on the school’s 
performance, the aim is to identify responsibility for what can be changed in the 
way the school operates, rather than looking at the differences that are specific to 
the children and their external circumstances. This focus enables the SNED method 
to identify the extent of the school’s responsibility for pupils’ performance levels. 

Weighting the factors and indicators 

Recognition of the school’s responsibility for factors affecting performance that 
could be altered is not simply a rhetorical feature expressing a position of principle, 
but is based on statistical measurement. In order to measure school performance, 
indicators have been constructed for each of the six factors set out above. 



Table 2. Indicators associated with each factor and their weighting in 2000-2001 



Factors 


Weighting 


Indicators 


Effectiveness 


37% 


Performance ratings in Spanish and mathematics 
according to SIMCE. 


Improvement 


28% 


Differences in results obtained in Spanish and 
mathematics according to SIMCE. 


Initiative 


6% 


Regular technical educational activities for groups. 



optional additional training activities. 

Formation of a management team meeting once a month. 
Organization of work in networks. 

Establishment of pupils’ group meeting at least once a month. 
Monitoring of pupils with learning difficulties as part of 
general education and/or work activity. 



Better working 2% 


Classification of the school according to the Ministry of 


conditions 


Education’s inspection system. 


Equality of 22% 


Pupil pass ratio. 


opportunity 


Pupil retention ratio. 



Mixed groups in operation. 

Aim of ensuring that the whole of the school-age population 
attends school, absence of discriminatory practices, for 
instance expulsion or cancellation of enrolment of pupils 
needing to repeat a year or of pregnant pupils, or expulsion 
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of pupils during the school year; acceptance of pupils with 
multiple or severe disabilities. 

Involvement of 5% Existence of a teachers’ council. 

teachers, parents Establishment of parents’ and guardians’ groups. 

and guardians Pupil participation by means of a pupils’ group. 

Setting of management and educational goals. 

Degree of involvement of parents and guardians. 
Willingness of parents and guardians to assist in 
the educational process. 

Parents’ views. 

Teachers’ views. 

Source: Ministry of Education, February 2000. 



Table 2 shows quite clearly that academic achievement indicators are not the only 
factor taken into account in the evaluation of a school’s performance. Different types 
of indicators, obtained from different sources, are used to provide a fuller and fair- 
er picture. Some are obtained by testing, others from surveys and reports, and regular 
statistics are standardized to form indicators. 

SNED is mainly based on information compiled at the national level. Its sources 
of information are: (i) SIMCE for academic performance in language and mathe- 
matics, and surveys of parents and guardians; (ii) the Ministry of Education for 
reports of schools inspectors; (iii) the Ministry of Education for enrolment statistics 
and pass and fail rates. Some indicators, however, require information that is avail- 
able at the provincial level, and a SNED survey has been devised for this administrative 
level which is accompanied by a SNED school file, both being managed by the region- 
al provincial directors of education. 

SNED is a flexible system. Weighting and the indicators themselves vary over 
time. There are two main reasons for this: changes are introduced firstly with a view 
to developing more sophisticated ways of measuring policy orientations and sec- 
ondly to ensure that the variables involved maintain their differential capacity. 

The scores obtained are subsumed in a single numerical value that is known 
as the SNED rating, which establishes each school’s position within each homoge- 
nous group, thus establishing a ranking order among them. Schools ranked in the 
top 25% in each group are then identified and if they come into the top 25% in their 
region they are designated as outstanding and receive the corresponding financial 
recompense. 

Financial aspects 

Between 1990 and 1999, the education budget tripled. SNED is a way of imple- 
menting a policy of differentiated remuneration for teachers. 

“SNED rewarded 1,699 schools for their outstanding performance in 2000- 
2001. Although this is 33% lower than the total for 1996-97, the schools themselves 
are now bigger. The number of teachers involved has risen from 30,600 to 32,600 
and the average roll has risen from 396.7 to 452.3. This sheds light on the current 
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debate about the ideal size of schools. The amount of the bonus awarded to each 
teacher has increased by 24% and the budget for the system has increased by 32% 
in constant terms. 

Conclusions 

The impact of SNED will be fully evaluated only as of March 2001. Nevertheless, 
preliminary evaluations of its first two applications have been carried out (1996/97 
and 1998/99, Chile, Ministry of Education, 2000), and they already make it possi- 
ble to identify some of the effects SNED has had. 

One concerns knowledge and the acceptance of the system. In evaluation this 
is central. The evaluations carried out show that in the few years it has been applied, 
SNED has become known, valued and accepted by school heads and teachers. This 
is important for two reasons. Firstly, the system seeks to improve school manage- 
ment by rewarding the introduction into the school’s everyday life of management 
methods that are in keeping with educational policy. Thus, as well as rewarding 
high-quality teaching SNED is also an instrument for improving management by 
school heads. Secondly, acceptance of SNED implies that an evaluation culture has 
been strengthened and that this type of evaluation is becoming an instrument for 
change in the school, since it promotes changes in behaviour. Another aspect to be 
noted is that the Colegio de Profesores (teaching union) has accepted the system, 
albeit with some reservations. This is significant, since it represents a departure from 
the union’s traditional stance in the debate as to whether rates of pay should be stan- 
dardized or diversified. The union’s present position is to accept diversification to 
the extent that evaluation is seen as recognition of professionalization. 

From the management point of view, SNED has two main effects. One is to 
encourage decentralization, as: (i) the attention of SNED is focused on processes 
and levels of achievement in the individual school; (ii) rewards are allocated to indi- 
vidual schools; (iii) responsibility for performance devolves to the school; and (iv) the 
school is strengthened. Secondly, the system involves a change in management style 
from input management to process and performance management. 

Notes 

1. This text is the exclusive responsibility of its author and does not commit UNESCO. 

2. During the 1990s, practically all the countries in Latin America started educational 
reforms centred on the institutional framework. 

3. See Hanushek, Rivkin & Kain (1999); Figlio (1997); Ballou Podgursky (1999). 

4. A more detailed analysis of the military regime’s education policy may be found in 
Espmola 6c de Moura Castro, 1999. 

5. The Ministry of the Interior is responsible for public order, security and social peace, 
and for the civil administration of the State at regional level. Under the military regime, 
mayors (and consequently those responsible for municipalized education) and school 
heads were appointed by the central authority of the Ministry of the Interior, and there- 
fore were accountable to that authority. This changed in 1991. After that date mayors 
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were elected democratically and school heads were appointed by competitive exami- 
nation. 

6. A ‘deconcentrated’ system is regarded as having become ‘decentralized’ when the author- 
ities that govern it — in this case the mayors — are no longer appointed by the central 
authority but are democratically elected. 

7. The USE is a kind of voucher applied at the national level. For an analysis of the effi- 
ciency and effectiveness of the system, see McEwan &c Carnoy, 2000. 

8. These amounts are given in 1996 pesos. Calculations by McEwan 8c Carnoy, op. cit., 
US $1 = (approx.) CLP $400 in 1996. Thus, 1 USE is the equivalent of US $20 (1988) 
and US $30(1996). 
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S C HOC L AUTONOMY AND E V A L U AT I O N 

SCHOOL AUTONOMY AND 
ASSESSMENT IN MEXICO 

Sylvia Schmelkes 



Decentralization and school autonomy 

The modern education system in Mexico emerged as a consequence of the revolu- 
tion (1910-21). The Ministry of Education (Secretaria de Educacion Piiblica) was 
founded in 1921. Though it began to function in a decentralized mode, it soon 
realised centralization was necessary to insure provision of educational services, par- 
ticularly in rural areas (Arnaut, 1998). 

The system remained centralized until very recently. In 1992, with the signa- 
ture of the National Agreement for the Modernization of Basic and Teacher-Training 
Education (Secretaria de Educacion Publica, 1992) by the President of the Republic, 
the union, and the governors of the states of the federation (Mexico has thirty-one 
states and one federal district), the administration of basic and normal education 
was decentralized to the state level. The federal government retained control of cur- 
riculum, evaluation and compensatory measures to ensure equity in the provision 
of educational services among the states. 

Mexican basic education experienced a noteworthy expansion during the 1960s 
and up to the end of the 1980s. It was designed as a vertical system and expanded 
following this model. Until very recently, the Mexican education system had had 
no experiences in school autonomy. Schools were conceived as units where uniform 
instructions emerging from the centre were to be strictly followed. There was, the- 
oretically, little room for decision-making at the school level. 
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This situation has changed little with decentralization. The focus of power has 
perhaps shifted somewhat, from the central to the state level. But the system is still 
vertical in nature, and schools are supposed to be supervized on how they follow 
instructions. 

In spite of this, progress has been made towards increasing the autonomy of 
schools experimentally and under close control of project designers. This is a con- 
sequence of growing evidence of enormous diversity of schools in Mexico (Rockwell, 
1995; Schmelkes, 1999) as well as the importance of schools as institutions and their 
role in educational improvement. ‘ 

I personally had the opportunity of directing an intervention project in the state 
of Coahuila in 1996 aimed at developing the capacity for school planning in 200 
preschools and primaries. In 1998, 1 acted as advisor to the state of Nuevo Leon for 
the development of a project aimed at preparing primary school supervisors for fos- 
tering and supporting educational planning in the schools under their jurisdiction. 

In the federal district, schools are invited to draw up ‘school projects’ as col- 
lective endeavours of the school team, aimed at improving the quality of learning. 
In 1996, a school management project designed centrally and aimed at improving 
the capacity of schools for planning their academic activities began in 200 primary 
schools in four states. The experience has been relatively successful and has grown 
steadily. To date, 1,697 of the country’s primary schools are participating, in nine- 
teen of the thirty-one states (Secretaria de Educacion Publica, 2000^). 

One of the most important programmes of the new administration in educa- 
tion^ is called ‘quality schools’. This programme is aimed at urban schools located 
in marginal areas that volunteer to participate in a school improvement programme. 
School projects designed by school teams with the participation of the community 
compete to obtain financial resources aimed at improving and increasing educational 
infrastructure, equipment, didactic materials and other special educational needs. 
Participating schools are closely monitored, supported during the process and eval- 
uated at the end of the school year to determine whether the school is eligible to 
continue within the programme. 

The curricular reform for the initial formation of teachers in 1997 includes a 
course on school management for the first time in history.^ Also, a nation-wide course 
is being offered for principals and supervisors dealing with school management and 
school autonomy."* 

As can be seen, the system is cautiously moving towards increasing school 
autonomy. There is no doubt as to where the trend is leading to in the near future. 
The caution with which Mexico is moving in this direction is due to fear of the actu- 
al risks of increasing inequality due to the incapacity of the education system to 
adequately support schools in their exercise of autonomy. 

Assessment: the generation of information 
and its use^ 

The Mexican Ministry of Education has vast experience in the assessment of learn- 
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ing outcomes. After Chile, which was the first country in Latin America to system- 
atically assess the learning outcomes of its students, Mexico is perhaps the country 
in Latin America with the most experience in educational assessment, since it has 
been systematically assessing students since 1970.^ During these three decades, the 
expertise of the Mexican evaluators has increased and the quality of both assess- 
ment instruments and analysis has improved. During the previous administration, 
the Directorate for Evaluation within the ministry carried out several important 
learning assessment studies and participated in a number of international evalua- 
tions. There is, therefore, a fair amount of information regarding Mexican students’ 
learning outcomes in basic education. 

Paradoxically, however, these efforts have been of little use in informing deci- 
sion-making at the different levels of educational administration. Educational 
authorities are either unaware that this information exists, or they do not consider 
it relevant for their policy-making and activities. Also, these results are not widely 
published. Society at large receives no information on the state of the education sys- 
tem, despite the General Law of Education that requires the dissemination of 
evaluation results.^ There has been a growing public demand for this information 
to be made public.® The new administration has been receptive to these demands 
and has expressed interest in creating a relatively autonomous National Institute for 
the Evaluation of Basic Education. This institute would include an area dedicated 
to the production of indicators regarding progress of education in general, a second 
area dedicated to the evaluation (and self-evaluation) of educational institutions, 
and a third area for the assessment of student achievement in basic education. One 
of the main purposes of this institute would be to make evaluation results available 
to the general public, thus the desire for its relative autonomy from the Ministry of 
Education (Rangel Sostmann, 2000). 

The future trend, then, also seems to be clear from recent educational devel- 
opments in the country. More importance will be given to evaluation and assessment; 
evaluation and assessment results will be made public; presumably, the use of eval- 
uation results for educational decision-making will qualitatively increase. 

Innovations in autonomy 

I will refer here to two innovations promoting school autonomy: the national school 
management project, which I briefly described above, and the project I participat- 
ed in in the state of Coahuila. 

The national school management project’s objective is to test a strategy for 
training and fostering teamwork in schools. Teams of teachers, with the principal as 
leader, are encouraged to diagnose the national educational situation at the prima- 
ry school level, to identify specific problems that hinder the achievement of goals, to 
develop a school project aimed at overcoming those problems, and to evaluate the 
results obtained. A technical group at the state level was trained specifically to sup- 
port schools in this process. Participation in the project was voluntary. The diagnostic 
exercise is carried out utilizing school statistics, analysing examination results, revis- 
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ing the students’ notebooks, registering the use of time in school, and collecting and 
analysing the opinion of students and parents. Training of the school team starts with 
a review of the curriculum and of educational materials provided by the government 

One way of developing teamwork, especially favoured by the school manage- 
ment project, is through strengthening the school council, which is composed of the 
headmaster and all of the teachers in the school. The school council is where school 
planning ought to take place, where discussions and consensus-building are most 
likely to occur, and where school personnel acquire the necessary training to carry 
out the transformations. The project discovered serious limitations of teachers and 
headmasters in conducting school council meetings. Teachers were not accustomed 
to following an agenda, to making use of the floor in an orderly manner, and to 
reaching a consensus. It was determined that training was needed for running a 
school council. On the other hand, school councils meet only once a month for a 
period of two and one half hours — obviously not enough time to reach consensus 
on goals and strategies. The project is aware that more time is required in the daily 
school routine for teamwork building. 

No evaluation of this project has been published, though it is known that infor- 
mation on its progress has been collected systematically. Nevertheless, there are 
several indications that the project has had an important impact on the education 
system nationally. Authorities have developed an interest in the school as an insti- 
tution. They have approached the schools more frequently and with more academic 
objectives than before. In many states, hindrances to the project (such as a merely 
regulating or controlling presence of the supervisor, or an excessive administrative 
load on both principals and teachers) have been gradually modified. In many cases, 
material needs of schools have been attended to.^ 

One of the most important results is the existence of well-trained consulting 
teams in all the participating states. These teams are aware of the problems in the 
day-to-day operation of the schools. They are well versed in the educational pur- 
poses of the primary school, as well as in the approach and content of the curriculum 
and of other materials and projects aimed at improving student learning. They have 
developed differentiated advising strategies aimed at strengthening work in the class- 
room, in school management, and in the relationship between the school and the 
community. The diagnostic exercise has proven to be the main tool for provoking 
reflection among the school personnel regarding difficulties in teaching, organiza- 
tion, and the relationships with parents, as well as the main causes of these problems. 
A little over 90% of the schools have designed a school project as a result of an 
analysis of the school’s problems.'^ 

Most school principals have become involved in the academic discussion of 
schoolwork. Most schools have also improved the relationship between school and 
the parents. Principals in participating schools have spoken well of the project in 
the regional technical councils and have thus fostered the project’s growth. 

A strategy has been defined for the generalization of the project to all complete 
primary schools, excluding those that are benefited by special compensatory pro- 
grammes, in states that have participated in it for three years. 
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The following conditions have been identified as impinging on the adequate 
development of the project: 

® The project has discovered many activities that interrupt schoolwork. 

® Lack of or insufficient involvement of principals and supervisors. The school 
project progresses more visibly and solidly when principals and supervisors 
are involved. 

® The administrative load and the number of programmes that distract the super- 
visors, principals and teachers from their activities. 

® The time allotted for collective work within the school is clearly insufficient 
for the design and monitoring of a school project. 

® The rotation of teachers and/or their arrival at the school later than the begin- 
ning of the school year hinders the integration of workable and stable teams. 
® The lack of co-ordination between the offices and departments that develop 
projects that are supposed to help schools results in an excessive demand for 
training, supervision and writing of reports or evaluations. 

Thus, this project, as well as the reflections that have accompanied its development, 
seems to show that more school autonomy is both possible and beneficial in the 
Mexican education system. It also shows, however, that the structure of the educa- 
tion system requires important transformations to allow for the effectiveness of 
school autonomy. 

The project in Coahuila works mainly with the notion of school planning. The 
annual plan, the School Project, the concept of school planning itself, are not only 
old concepts, but they are in continuous use within our school system. The inten- 
tion of this project was, precisely, to relate it to school change. From this perspective, 
authors such as Maden and Tomlinson (1991) and MacGilchrist et al. (1995) find 
in school planning, when it occurs under certain conditions, an effective instrument 
for improving school quality. 

I wish to note three specific characteristics that make this instrument different 
from traditional planning and that are reflected in the project I am reporting: 

® The School Project is an instrument for changing the school. It plans change, 
not routine (the annual plan, in our schools, plans routine). This is based on 
a common-sense premise: Quality improvement must recognize that problems 
exist. If we continue to do the same things, the problems will persist. To solve 
the problems, we must change what we do. 

® The School Project is understood as the product of the collective action of the 
school community: of the teaching staff, under the leadership of the school 
principal, and after consultation with and approval of the school’s communi- 
ty. This is why the School Project also becomes an instrument for professional 
development. 

• The School Project starts being useful the moment it is designed. The School 
Project is more than a document for educational authorities; it is a plan to ori- 
ent daily school activities in a defined direction. 

The study that accompanies this intervention project shows that, in spite of our 
aims, schools that are able to plan their change to achieve improved learning are 
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still the exception. Only about one-third of the schools actually took initial steps 
towards school change and improvement. The study is rich in information on the 
obstacles for achieving this aim. These obstacles have to do with the teaching cul- 
ture, with the way schools function on a day-to-day basis, and the culture and 
structure of each of the schools in the sample. Other obstacles have to do with the 
way the larger education system works. The main obstacles encountered include: 

© The culture of simulation. This seems to be the product of many years of use- 
less work required of teachers and principals. A number of schools work towards 
improving quality, without defining clear purposes or assessable commitments. 
Though we have little information up to now about the impact on schools of 
the School Project, we can say that in this case the School Project has a nega- 
tive impact. 

® The lack of leadership of school principals. Many projects are one-person pro- 
jects, though a number of them are presented as if they were collective. The 
development of the projects shows that teachers do not comply with the com- 
mitments set out at the beginning of the school year. Principals require that 
project training reaches teachers, and they ask for more frequent visits by the 
technical group, seemingly to validate their request to the teachers to partici- 
pate in the development of a school project. 

© Training flaws and gaps. Even in cases in which a collectively owned project 
emerges, it is limited by the knowledge and training of both the principal and 
the teachers. This reduces the diagnoses of the true causes of learning prob- 
lems, or of solving the problems that are diagnosed. The implications of this 
finding for the development of educational decentralization to the school level 
are extremely important because they address the relevant role the state — at 
either central or state-level — must play to ensure that such a process produces 
a greater quality of learning. 

® The fourth obstacle is related to an important aspect of the teaching culture, 
which makes the teachers tend to seek the causes for learning problems out- 
side the classroom and the schools. It is true that a number of external factors 
help explain learning difficulties. However, the fact that the factors found in 
the school and in teacher behaviour are not taken into account may lead to 
actions that only produce frustration and that will hinder future attempts at 
school change. 

® The fifth obstacle has to do with the larger education system. It is a system that 
operates in a vertical top-down manner and that, in general, appears to be dis- 
tant from the school. With a few exceptions, the supervisor does not fulfil 
his/her role adequately and his/her reputation is clearly deteriorated among 
school staff: Nothing is expected from him/her on the part of teachers and 
headmasters. Many schools are in such physical deterioration, or with such an 
evident lack of infrastructure — mainly explained by national economic condi- 
tions that give preference to salaries over infrastructure — that the temptation 
of dedicating the School Project to solve these problems is enormous. The 
request for greater counselling and training, as well as for more frequent vis- 
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its on the part of the technical group, are indications of an education system 
that operates bureaucratically without regard to the school’s needs. It is diffi- 
cult to understand greater autonomy to the schools while this manner of 
operating of the education system remains. 

In spite of all these difficulties, we can end this section on an optimistic note. In one- 
third of the primary schools in the sample one can observe some trends towards 
important change or transformation. Three of these are particularly relevant: 

® Schools begin to think in terms of students learning. School projects in gener- 
al are related to problems directly linked to learning. The central activity of 
the school appears to be revolving around the education of the students. 

® Teachers work collectively and discover the value of teamwork and the value 
of other teachers. In one-third of the schools mentioned frequent meetings of 
the School Technical Council are held during which academic issues are dis- 
cussed because the School Project is part of the agenda and is monitored in 
these meetings. More important, however, are indications that the discovery 
of the value of teamwork seems to be irreversible. A similar trend was observed 
in the few school districts in which periodic meetings of the District Technical 
Council were held. Here the headmasters discovered the value of the inter- 
change of experiences, the role of other principals as counsellors of each other’s 
projects, and the importance of dealing with academic issues. In these cases, 
the supervisor transformed his role and was re-valued by principals. 

® School approaches the community. This is perhaps the most visible change as 
a consequence of the project. Since schools are recommended to consult par- 
ents on the School Project, after which some schools ask for specific 
commitments on their part, both principal and teacher discover that parents 
may become very important allies in the child’s education. Myths in Mexican 
school culture relative to parents — that they are apathetic, uninterested, une- 
ducated — are destroyed. The school becomes nearer to the community and 
takes it more into account. This is another process that might be irreversible. 
There are two ways in which the relationship between the school and the com- 
' munity has changed. One refers to the possibility of discussing problems that 
underlie existing conflicts between the school and the community or the par- 
ents. An example is a school that had no fence, exposing the children to the 
risk of accidents in the street. Working together, the school and community 
found the resources to build the fence, and the parents provided the labour to 
build it. This changed the relationships between school and community that 
led to other collaborative projects. Another way is the participation of parents 
in the fostering of their children’s progress in school. In several schools, teach- 
ers were able to state clearly what they expected from the parents: less unjustified 
absenteeism, the reservation of a special time and space for doing homework, 
and the provision of an adequate, nutritious meal before school for the chil- 
dren. The parents became more aware of what was happening in school and 
of their own responsibility towards their children’s progress. 

The changes noted above only occur in one-third of the schools in the sample. 
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Nevertheless, the fact that they are present seems to demonstrate that change that 
stems from the school is possible. On the other hand, they indicate that the trans- 
formation we are after is a slow process that must begin with building the scaffolding 
that will permit an improvement of the quality of the school in the future. This, 
however, must occur at the same time the system is transformed. A small indication 
of this change was seen in two of the school districts we worked in, though this is 
not analysed in this paper. The idea is for the system to position the individual school 
at the centre of its activities and for it to function around its needs and weakness- 
es, as well as its strengths. 

Innovations in assessment’^ 



During the previous administration (1994-2000), an important evaluation of stu- 
dent achievement with respect to national standards for primary education in reading 
and mathematics was carried out.^^ Criterion-referenced tests were developed for 
the six grades of primary education in both areas, and were applied to a national- 
ly representative sample of different types of schools: urban private schools, urban 
public schools, rural public schools,*^ Cursos Comunitarios,*'’ and Indian bilingual- 
bicultural schools. 

In the following, I refer to main conclusions from the results of the applica- 
tion carried out in May 1998. 

Most Mexican students meet national standards or are near meeting them. 
Nevertheless, about one-third of the students are far from meeting national stan- 
dards, and these grow from the fourth grade onwards. 

However, schools are heterogeneous regarding the achievement of national 
standards in both reading and mathematics. With few exceptions, there are signif- 
icant differences in students’ scores among the different types of schools, which 
reflects the weight of socio-economic and cultural factors on student achievement. 

The exceptions to the previous conclusion, however, are interesting: differences 
in reading tend to disappear after the third grade between rural schools and Cursos 
Comunitarios, and in mathematics, between Cursos Comunitarios and both rural 
and urban public schools. 

There are also important differences in the proportion of students achieving 
standards among the different types of schools. Differences between private and 
public urban schools tend to grow as one progresses in the school grades. The oppo- 
site occurs with the differences between urban and rural public schools, which tend 
to become smaller as one progresses in the school grades. 

An analysis by quartiles and deciles, however, demonstrates the existence of 
strong differences between schools of the same type. The analysis of deciles also 
shows that there are good schools in all populations, from private urban schools to 
indigenous schools,^^ and that a good school is just as good in one population as in 
another. In several cases, good indigenous schools have a greater proportion of stu- 
dents achieving standards than good private schools. This, of course, is a very 
important result, because it demonstrates that excellent schools in any of the cir- 
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cumstances and populations studied exist and, therefore, more can be developed. 

This evaluation based on national standards for primary schools in reading 
and mathematics is considered an innovation for two reasons. The first concerns 
how the tests were constructed: it is possible not only to give an idea of the degree 
of development of basic skills among the primary school students, but also of being 
able to say something about what the students know and what they lack in order 
to meet the standards. For example, in the case of reading, it is clear that in gener- 
al students are able to read sentences and paragraphs, but those that do not meet 
the standards cannot integrate the meaning of a complete text (three or more para- 
graphs, depending on the grade). The second reason concerns the type of analysis 
performed: one of the main objectives is the measurement of inequality in students’ 
learning results between different types of schools. Also, the identification of excep- 
tional schools and of schools presenting severe problems is possible. This evaluation 
was accompanied by contextual information on the child, the family and the school, 
which has not yet been analysed, but which will allow for the hypothetical identi- 
fication of conditions in which the probabilities of having a we 11 -performing school, 
or of having a poorly performing school, become stronger. The relevance of this 
type of analysis for policy making at the different levels is evident. 

Towards a fruitful relationship^^ 



The quality of primary education, as measured by student achievement, is unequal- 
ly distributed among different types of schools in Mexico. In general, the poorer the 
context and the students and their families, and/or the more the children belong to 
families from cultural (ethnic and linguistic) minorities, the lower the achievement 
of the students and the proportion of students achieving national standards. 

This occurs within a system that functions vertically and that expects schools 
to follow instructions, not to make decisions. However, the evaluation that we have 
described clearly identifies schools in all categories with excellent learning outcomes 
among their students. Excellent schools in the different categories have statistically 
equivalent learning outcomes. It is clear that these schools are the exception and not 
the rule. Nevertheless, it is also obvious that these schools are making their own 
decisions, and probably they are taking them collectively. Only this can explain their 
extraordinary results. 

Thus, decision-making within the school — or a certain and perhaps increasing 
degree of school autonomy — is undoubtedly advisable. 

Nevertheless, in a system in which inequalities in learning outcomes are as 
severe as we have described, autonomy on its own could become a further cause of 
inequality. The capacity of schools for diagnosing their problems, setting collective 
objectives, obtaining clear commitments from the different members of the com- 
munity, and monitoring and evaluating their progress is, as we have seen in the 
second section of this article, also unequally distributed. 

It would, therefore, be a mistake to conclude that the system may be weak- 
ened because schools can be made accountable for their results, as has occurred in 




I 

! 




584 



Sylvia Schmelkes 



England, where local educational authorities were practically done away with and 
the system favoured inspection and evaluation in place of local support. On the con- 
trary, the education system has to become stronger. Aided with results of evaluation 
studies such as the one we have mentioned, the system has to be able to support 
schools according to their capacity to make the right decisions. The education sys- 
tem should, therefore, use the results of student and school evaluations to identify 
both individual schools and contextual and school conditions that indicate situa- 
tions in need of special pedagogical and managerial support. Interaction between 
schools that have been identified as well performing, and those that have difficul- 
ties in achieving educational standards, could perhaps become one of the strategies 
of support. 

A fruitful relationship among non-punitive assessment, the fostering of school 
autonomy, the differential support to the schools that need it the most on the part 
of the education system, and the requirement of accountability on the part of the 
school, would thus seem the most sensible course for an education system charac- 
terized by inequality such as the Mexican case that we have described here. This, of 
course, requires qualitative leaps on the part of both the schools and the education 
system. Schools must put the student, and his/her learning, at the centre of their pur- 
suit. The system must place each school, according to its conditions, problems and 
potential, at the centre of its activity. 

Notes 

1. 1 refer to the well-known and growing body of literature on school effectiveness and 
school improvement. 

2. The present administration took office 1 December 2000, For the first time in seventy- 
one years, elections were won by an opposition party. Cf. Rangel Sostmann, 2000, 

3. Secretaria de Educacion Publica. 1997. The course is also offered in the programme for 
the initial training of pre-school and lower secondary schoolteachers. 

4. Secretaria de Educacion Publica. 2000f?. The course is also offered for secondary school 
principals and supervisors. 

5. Part of this section is based on Schmelkes, 2001. 

6. There is a programme for the promotion of teachers that includes an evaluation of their 
students. As part of this programme, around 7 million students are tested yearly in pri- 
mary and secondary schools. There is an on-going longitudinal study for the evaluation 
of primary schools that has collected data in the same schools for five years. A diag- 
nostic evaluation of students entering secondary schools (an aptitude test) has been 
carried out for ten years in certain cities within the country. A national evaluation based 
on both primary school standards for reading and mathematics has been carried out 
with a nationally representative sample of schools for three consecutive years, and for 
the first time in secondary schools. Mexico has participated in the following interna- 
tional studies: The TIMSS and the TIMSS-R, the international assessment of third-grade 
students co-ordinated by the Latin American and Caribbean Regional Office of UNESCO 
(Laboratorio Latinoamericano de Evaluacion de la Calidad) and, more recently, the 
PISA project of OECD. 

7. Article 31 of the General Law of Education states: ‘Educational authorities will inform 
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teachers, students, parents, and society at large, of the results of evaluations and of 
other information that allows for the measurement of the development and progress of 
education in each state’. Secretarfa de Educacion Publica, 1993. 

8. Cf., for example, Observatorio Ciudadano de la Educacion, 1999. Also in: wvw.obser- 
vatorio.org. 

9. In Mexico, textbooks for primary school students are free. There are also books for the 
teacher, cards for the development of mathematical competences, and a classroom 
library. It has been found that in many cases these materials are not used, or not suffi- 
ciently exploited. 

10. The project excludes schools that have more than one class per teacher. Many schools 
are ‘multigrade’. It is not easy to have a reliable statistic of these schools, but 25% of 
all primary schools had only one teacher in 1998/99. Ezpeleta 8c Weiss, 2000. 

11. What follows is based on Schmelkes, Forthcoming. 

12. During the last year of the administration, the same exercise was carried out for lower 
secondary education also. However, in this article I refer only to results at the prima- 
ry education level. 

13. There are practically no private rural schools in Mexico. Private primary schools rep- 
resent 5% of the total number of schools in the country, and give service to approximately 
10% of the primary school students. 

14. Cursos Comunitarios is a form of supplying primary education to very small rural com- 
munities (under thirty school-age children). There is one ‘para-teacher’ (secondary school 
graduate) for the three levels (instead of six grades) of primary education: 

15. Indigenous schools offer bilingual education to primary school children of the sixty- 
two different ethnic and linguistic groups in the country. 

16. What follows is based on Secretaria de Educacion Publica, 1999. 
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Historical antecedent 

Learning and civilization have flourished during many eras of Afghan history. A 
cradle and crossroads of ancient civilizations, Afghanistan has contributed to enrich- 
ing the culture of the region in which it is located throughout its history. During the 
Islamic period, many centres of learning were established in cities such as Balkh, 
Herat and Ghazni, which produced scholars in philosophy, literature and science 
to serve the people of the region and beyond. The tenth-century medical treaties 
written by Ibn-i-Seena Balkhi were used in universities until the end of nineteenth 
century. Many Afghan poets and philosophers have enriched human thought and 
civilization. These include the tenth-century Abu Rayhan Biruni, eleventh-century 
Hakim Sanai, thirteenth-century Khwaja Abdullah Ansari and Maulana Jalaludin 
Roumi, and the seventeenth-century Kushal Khan Khattak and Rahman Baba. The 
nineteenth-century Afghan political philosopher Jamaludin Afghani, who had many 
disciples, travelled through Asia, Africa and Europe to promote national indepen- 
dence and Islamic solidarity and to explain the relation between Islam, science and 
progress. 
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At the beginning of the twentieth century, the Islamic tradition permeated every 
aspect of Afghan society, including education and training at all levels. Education 
was provided at home, in the mosques, in informal schools linked to mosques, in 
religious centres (Madrasa) and in circles of scholars. These institutions were sup- 
ported by parents, local communities, religious and tribal leaders and through private 
resources. The Madrasa provided general knowledge and theological studies for 
young men and prepared religious and community guides and teachers. Some oppor- 
tunities for studies in areas such as writing, poetry and literature, history, science 
and traditional medicine were available through private tutoring and in small infor- 
mal circles. Mosques and related schools served as basic community centres in which 
pupils could study the Koran, learn about Islamic values and ethics, and acquire lit- 
eracy and numeracy. Traditional education also helped prepare young people (mainly 
boys) for work through informal apprenticeships in local arts and crafts, farming 
and commerce. 

Educational developments during the twentieth century reflected the religious 
and traditional character of Afghan society. Afghanistan’s changing political con- 
text, including the social and economic policies of the successive regimes and the 
public’s aspiration for the education of their children, influenced the nature and 
form of education and its expansion. In light of the multi-ethnic composition of 
Afghan society, cultural and linguistic policies were important factors in the devel- 
opment of education. Textbooks and educational materials were prepared in two 
national languages (Pashto and Dari). 

A number of factors have hindered educational development in Afghanistan. 
The country is landlocked. Its economy is agricultural (85% of the population live 
in rural areas) and its resources unexploited. Economic constraints have proven to 
be the major obstacles to its educational development. Its GDP per capita was at 
US$60 in 1960, US$160 in 1976, and estimated at about US$600 in 1995. Its domes- 
tic product increased from 12.5 billion Afs. in 1953 to 90.0 billion Afs. in 1976.' 
In the 1960s, the United Nations classified Afghanistan as one of the world’s twen- 
ty-five least developed countries. 

In the second half of the last century, the international organizations played 
an important role in stimulating educational developments worldwide by setting 
norms and targets and by providing technical and financial assistance. This inter- 
national co-operation contributed to the expansion and qualitative improvement of 
modern education in Afghanistan. 

Pre-Second World War developments 

The beginning of modern education in Afghanistan corresponds with the 1903 estab- 
lishment of the first secondary school, Habibia, in Kabul. Afghan and foreign teachers 
were recruited for Habibia, the aim of which was to train personnel for the civil ser- 
vice. Soon after, several modern primary schools and a teacher training institution 
were established. In 1909, the government set up a Board of Education to supervise 
education across the country, including: traditional educational institutions. The 
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board’s duties included that of approving school curriculum and textbooks. Following 
Afghanistan’s independence from the United Kingdom in 1919, King Ammanullah’s 
government made educational development a high priority and the first minister of 
education was appointed in 1922. During the 1920s, a number of primary and sec- 
ondary schools — including a school for girls and an adult education centre for 
women — were established in Kabul, along with several vocational schools in areas 
such as agriculture, arts and crafts and public administration. For the first time, 
Afghan boys and girls were sent abroad for studies. The establishment of diplomatic 
and cultural relations with foreign countries contributed to the development of a 
modern education system. Through bilateral co-operation with countries such as 
Turkey, France and Germany, foreign teachers and expertise as well as student schol- 
arships were made available to Afghans. 

Educational development experienced a set-back in 1929 when modern schools 
were closed during the nine months of civil strife and anarchy that followed the 
January abdication of King Ammanullah. When, following upon this period of con- 
flict, Nadir Shah became king of Afghanistan, the country’s thirteen primary and 
secondary schools were reopened and attention was again given to the development 
of education. A new constitution, adopted in 1931, made primary schooling com- 
pulsory for Afghans and placed all modern educational institutions under the control 
of the State. (Traditional educational entities remained independent.) Education at 
all levels — primary through tertiary — was provided free for Afghans. In 1933, Nadir 
Shah was assassinated and his son Zaher Shah came to the throne. 

The Afghan education system saw some expansion during the 1930s. Special 
attention was given to the development of Pashto as one of the two main national 
languages of Afghanistan. A number of secondary schools were established in the 
provinces and several traditional religious schools were modernized and incorpo- 
rated into the formal educational structure. A secondary school for girls was 
established in Kabul, along with two secondary vocational schools — one mechani- 
cal, the other agricultural — and a school for the training of assistant doctors and 



pharmacists. Turkish medical advisers and several Afghan doctors, who had trained 
abroad, aided in the creation in 1932 of the Faculty of Medicine — which was later 
to evolve into the University of Kabul, officially established in 1946. The Faculty of 
Medicine graduated the first group of eight intern doctors and twelve pharmacists 
in 1937. With a view to the social and economic development of Afghanistan, in 
1938 the government sent a group of secondary school graduates to universities in 
France, Germany and the United States. 

The pace of educational development in Afghanistan slowed down consider- 
ably during the Second World War — during which Afghanistan remained 
neutral — because of a variety of economic and technical constraints: most foreign 
experts and teachers left the country, and equipment and materials needed for edu- 
cational institutions could not be imported. Nonetheless, by the end of the Second 
World War, the Afghan education system had expanded. During the twenty-year 
period beginning in 1930, the number of students had risen from 1,590 to 95,300, 
4,350 of whom were girls. The number of teachers had increased from fifty-three 
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in 1930 to some 3,000 in 1950; they served 368 schools, including 308 primary 
schools, twenty-five lower secondary schools (middle schools), sixteen upper sec- 
ondary schools (lycees), four vocational schools, one teacher training school and 
seven formal religious schools. There were also 456 students, of which forty were 
female, in five institutions of higher education. By the middle of the twentieth-cen- 
tury, 6% of Afghan children, ages 6-12, were enrolled in primary schools, out of an 
estimated population of 11 million people. 

Post-Second World War developments 

After 1945, in the context of political reform, the beginnings of a short-lived demo- 
cratic movement and the return of a number of Afghan specialists from abroad, 
several important measures were taken to encourage educational development in 
Afghanistan, In 1947, the Ministry of Education was reorganized to ensure the fur- 
ther development of primary, secondary, technical and vocational and teacher 
education. At this time, more focus was also placed on the improvement and pro- 
duction of textbooks, provision of teaching equipment and materials and school 
construction (all of which fell within the ministry’s portfolio). To promote the expan- 
sion of education, increased authority and resources were delegated away from the 
central office and toward provincial offices throughout the country. 

From the early 1950s, the Afghan Government focused on the systematic expan- 
sion of education and the improvement of educational quality. It launched a number 
of development plans, financed through both internal resources as well as bilateral 
and multilateral technical assistance funds and loans from the international com- 
munity — including the United Nations and countries such as the United States, the 
former USSR and Germany. In the first of three five-year development plans (for the 
period 1956-61), 6.5% of the national budget was devoted to social services, with 
958 million Afs. earmarked for education. This plan focused principally on prima- 
ry schooling. 

In view of the country’s need for a trained workforce, the second five-year plan 
(for the period 1962-67) emphasized secondary, technical and higher education. In 
this plan, the share of the national budget for social services was increased to 11.2%, 
with 1,759 million Afs. earmarked for education. The third five-year plan (for the 
period 1968-73) paid special attention to qualitative improvements in teacher edu- 
cation and reflected a commitment to the balanced expansion of education at all 
levels. This time, 16.7% of the total nation budget was allocated to social services, 
with over 3,000 million (or 3 billion) Afs. targeted for education. While primary 
education expanded as a result of these development plans, it was secondary edu- 
cation that experienced the fastest growth — as a result of public demand, especially 
in the provinces. 

Following a coup d’etat in 1973, after which Afghanistan was proclaimed a 
republic and Mohammad Daoud (a former prime minister and cousin of King Zaher) 
became head of state, Afghanistan launched a seven-year economic and social devel- 
opment plan (for the period 1976-83^ to mm was to accelerate economic growth 
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Decentralization and school autonomy 

The modern education system in Mexico emerged as a consequence of the revolu- 
tion (1910-21). The Ministry of Education (Secretaria de Educacion Piiblica) was 
founded in 1921. Though it began to function in a decentralized mode, it soon 
realised centralization was necessary to insure provision of educational services, par- 
ticularly in rural areas (Arnaut, 1998). 

The system remained centralized until very recently. In 1992, with the signa- 
ture of the National Agreement for the Modernization of Basic and Teacher-Training 
Education (Secretaria de Educacion Publica, 1992) by the President of the Republic, 
the union, and the governors of the states of the federation (Mexico has thirty-one 
states and one federal district), the administration of basic and normal education 
was decentralized to the state level. The federal government retained control of cur- 
riculum, evaluation and compensatory measures to ensure equity in the provision 
of educational services among the states. 

Mexican basic education experienced a noteworthy expansion during the 1960s 
and up to the end of the 1980s. It was designed as a vertical system and expanded 
following this model. Until very recently, the Mexican education system had had 
no experiences in school autonomy. Schools were conceived as units where uniform 
instructions emerging from the centre were to be strictly followed. There was, the- 
oretically, little room for decision-making at the school level. 
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This situation has changed little with decentralization. The focus of power has 
perhaps shifted somewhat, from the central to the state level. But the system is still 
vertical in nature, and schools are supposed to be supervized on how they follow 
instructions. 

In spite of this, progress has been made towards increasing the autonomy of 
schools experimentally and under close control of project designers. This is a con- 
sequence of growing evidence of enormous diversity of schools in Mexico (Rockwell, 
1995; Schmelkes, 1999) as well as the importance of schools as institutions and their 
role in educational improvement.’ 

I personally had the opportunity of directing an intervention project in the state 
of Coahuila in 1996 aimed at developing the capacity for school planning in 200 
preschools and primaries. In 1998, 1 acted as advisor to the state of Nuevo Leon for 
the development of a project aimed at preparing primary school supervisors for fos- 
tering and supporting educational planning in the schools under their jurisdiction. 

In the federal district, schools are invited to draw up ‘school projects’ as col- 
lective endeavours of the school team, aimed at improving the quality of learning. 
In 1996, a school management project designed centrally and aimed at improving 
the capacity of schools for planning their academic activities began in 200 primary 
schools in four states. The experience has been relatively successful and has grown 
steadily. To date, 1,697 of the country’s primary schools are participating, in nine- 
teen of the thirty-one states (Secretaria de Educacion Publica, 2000a), 

One of the most important programmes of the new administration in educa- 
tion^ is called ‘quality schools’. This programme is aimed at urban schools located 
in marginal areas that volunteer to participate in a school improvement programme. 
School projects designed by school teams with the participation of the community 
compete to obtain financial resources aimed at improving and increasing educational 
infrastructure, equipment, didactic materials and other special educational needs. 
Participating schools are closely monitored, supported during the process and eval- 
uated at the end of the school year to determine whether the school is eligible to 
continue within the programme. 

The curricular reform for the initial formation of teachers in 1997 includes a 
course on school management for the first time in history.^ Also, a nation-wide course 
is being offered for principals and supervisors dealing with school management and 
school autonomy.'* 

As can be seen, the system is cautiously moving towards increasing school 
autonomy. There is no doubt as to where the trend is leading to in the near future. 
The caution with which Mexico is moving in this direction is due to fear of the actu- 
al risks of increasing inequality due to the incapacity of the education system to 
adequately support schools in their exercise of autonomy. 

Assessment: the generation of information 
and its use^ 

The Mexican Ministry of Education has vast experience in the assessment of learn- 
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ing outcomes. After Chile, which was the first country in Latin America to system- 
atically assess the learning outcomes of its students, Mexico is perhaps the country 
in Latin America with the most experience in educational assessment, since it has 
been systematically assessing students since 1970.^ During these three decades, the 
expertise of the Mexican evaluators has increased and the quality of both assess- 
ment instruments and analysis has improved. During the previous administration, 
the Directorate for Evaluation within the ministry carried out several important 
learning assessment studies and participated in a number of international evalua- 
tions, There is, therefore, a fair amount of information regarding Mexican students’ 
learning outcomes in basic education. 

Paradoxically, however, these efforts have been of little use in informing deci- 
sion-making at the different levels of educational administration. Educational 
authorities are either unaware that this information exists, or they do not consider 
it relevant for their policy-making and activities. Also, these results are not widely 
published. Society at large receives no information on the state of the education sys- 
tem, despite the General Law of Education that requires the dissemination of 
evaluation results.^ There has been a growing public demand for this information 
to be made public.^ The new administration has been receptive to these demands 
and has expressed interest in creating a relatively autonomous National Institute for 
the Evaluation of Basic Education. This institute would include an area dedicated 
to the production of indicators regarding progress of education in general, a second 
area dedicated to the evaluation (and self-evaluation) of educational institutions, 
and a third area for the assessment of student achievement in basic education. One 
of the main purposes of this institute would be to make evaluation results available 
to the general public, thus the desire for its relative autonomy from the Ministry of 
Education (Rangel Sostmann, 2000). 

The future trend, then, also seems to be clear from recent educational devel- 
opments in the country. More importance will be given to evaluation and assessment; 
evaluation and assessment results will be made public; presumably, the use of eval- 
uation results for educational decision-making will qualitatively increase. 

Innovations in autonomy 



I will refer here to two innovations promoting school autonomy: the national school 
management project, which I briefly described above, and the project I participat- 
ed in in the state of Coahuila. 

The national school management project’s objective is to test a strategy for 
training and fostering teamwork in schools. Teams of teachers, with the principal as 
leader, are encouraged to diagnose the national educational situation at the prima- 
ry school level, to identify specific problems that hinder the achievement of goals, to 
develop a school project aimed at overcoming those problems, and to evaluate the 
results obtained. A technical group at the state level was trained specifically to sup- 
port schools in this process. Participation in the project was voluntary. The diagnostic 
exercise is carried out utilizing school statistics, analysing examination results, revis- 
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ing the students’ notebooks, registering the use of time in school, and collecting and 
analysing the opinion of students and parents. Training of the school team starts with 
a review of the curriculum and of educational materials provided by the government 

One way of developing teamwork, especially favoured by the school manage- 
ment project, is through strengthening the school council, which is composed of the 
headmaster and all of the teachers in the school. The school council is where school 
planning ought to take place, where discussions and consensus-building are most 
likely to occur, and where school personnel acquire the necessary training to carry 
out the transformations. The project discovered serious limitations of teachers and 
headmasters in conducting school council meetings. Teachers were not accustomed 
to following an agenda, to making use of the floor in an orderly manner, and to 
reaching a consensus. It was determined that training was needed for running a 
school council. On the other hand, school councils meet only once a month for a 
period of two and one half hours — obviously not enough time to reach consensus 
on goals and strategies. The project is aware that more time is required in the daily 
school routine for teamwork building. 

No evaluation of this project has been published, though it is known that infor- 
mation on its progress has been collected systematically. Nevertheless, there are 
several indications that the project has had an important impact on the education 
system nationally. Authorities have developed an interest in the school as an insti- 
tution. They have approached the schools more frequently and with more academic 
objectives than before. In many states, hindrances to the project (such as a merely 
regulating or controlling presence of the supervisor, or an excessive administrative 
load on both principals and teachers) have been gradually modified. In many cases, 
material needs of schools have been attended to.^ 

One of the most important results is the existence of well-trained consulting 
teams in all the participating states. These teams are aware of the problems in the 
day-to-day operation of the schools. They are well versed in the educational pur- 
poses of the primary school, as well as in the approach and content of the curriculum 
and of other materials and projects aimed at improving student learning. They have 
developed differentiated advising strategies aimed at strengthening work in the class- 
room, in school management, and in the relationship between the school and the 
community. The diagnostic exercise has proven to be the main tool for provoking 
reflection among the school personnel regarding difficulties in teaching, organiza- 
tion, and the relationships with parents, as well as the main causes of these problems. 
A little over 90% of the schools have designed a school project as a result of an 
analysis of the school’s problems. 

Most school principals have become involved in the academic discussion of 
schoolwork. Most schools have also improved the relationship between school and 
the parents. Principals in participating schools have spoken well of the project in 
the regional technical councils and have thus fostered the project’s growth. 

A strategy has been defined for the generalization of the project to all complete 
primary schools, excluding those that are benefited by special compensatory pro- 
grammes, in states that have participated in it for three years. 

1 3 6 
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The following conditions have been identified as impinging on the adequate 
development of the project: 

o The project has discovered many activities that interrupt schoolwork. 
o Lack of or insufficient involvement of principals and supervisors. The school 
project progresses more visibly and solidly when principals and supervisors 
are involved. 

o The administrative load and the number of programmes that distract the super- 
visors, principals and teachers from their activities. 

® The time allotted for collective work within the school is clearly insufficient 
for the design and monitoring of a school project. 

Q The rotation of teachers and/or their arrival at the school later than the begin- 
ning of the school year hinders the integration of workable and stable teams. 
® The lack of co-ordination between the offices and departments that develop 
projects that are supposed to help schools results in an excessive demand for 
training, supervision and writing of reports or evaluations. 

Thus, this project, as well as the reflections that have accompanied its development, 
seems to show that more school autonomy is both possible and beneficial in the 
Mexican education system. It also shows, however, that the structure of the educa- 
tion system requires important transformations to allow for the effectiveness of 
school autonomy. 

The project in Coahuila works mainly with the notion of school planning. The 
annual plan, the School Project, the concept of school planning itself, are not only 
old concepts, but they are in continuous use within our school system. The inten- 
tion of this project was, precisely, to relate it to school change. From this perspective, 
authors such as Maden and Tomlinson (1991) and MacGilchrist et al. (1995) find 
in school planning, when it occurs under certain conditions, an effective instrument 
for improving school quality. 

I wish to note three specific characteristics that make this instrument different 
from traditional planning and that are reflected in the project I am reporting: 

® The School Project is an instrument for changing the school. It plans change, 
not routine (the annual plan, in our schools, plans routine). This is based on 
a common-sense premise: Quality improvement must recognize that problems 
exist. If we continue to do the same things, the problems will persist. To solve 
the problems, we must change what we do. 

® The School Project is understood as the product of the collective action of the 
school community: of the teaching staff, under the leadership of the school 
principal, and after consultation with and approval of the school’s communi- 
ty. This is why the School Project also becomes an instrument for professional 
development. 

® The School Project starts being useful the moment it is designed. The School 
Project is more than a document for educational authorities; it is a plan to ori- 
ent daily school activities in a defined direction. 

The study that accompanies this intervention project shows that, in spite of our 
aims, schools that are able to plan their change to achieve improved learning are 
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still the exception. Only about one-third of the schools actually took initial steps 
towards school change and improvement. The study is rich in information on the 
obstacles for achieving this aim. These obstacles have to do with the teaching cul- 
ture, with the way schools function on a day-to-day basis, and the culture and 
structure of each of the schools in the sample. Other obstacles have to do with the 
way the larger education system works. The main obstacles encountered include: 

® The culture of simulation. This seems to be the product of many years of use- 
less work required of teachers and principals. A number of schools work towards 
improving quality, without defining clear purposes or assessable commitments. 
Though we have little information up to now about the impact on schools of 
the School Project, we can say that in this case the School Project has a nega- 
tive impact. 

® The lack of leadership of school principals. Many projects are one-person pro- 
jects, though a number of them are presented as if they were collective. The 
development of the projects shows that teachers do not comply with the com- 
mitments set out at the beginning of the school year. Principals require that 
project training reaches teachers, and they ask for more frequent visits by the 
technical group, seemingly to validate their request to the teachers to partici- 
pate in the development of a school project. 

® Training flaws and gaps. Even in cases in which a collectively owned project 
emerges, it is limited by the knowledge and training of both the principal and 
the teachers. This reduces the diagnoses of the true causes of learning prob- 
lems, or of solving the problems that are diagnosed. The implications of this 
finding for the development of educational decentralization to the school level 
are extremely important because they address the relevant role the state — at 
either central or state-level — must play to ensure that such a process produces 
a greater quality of learning. 

® The fourth obstacle is related to an important aspect of the teaching culture^ 
which makes the teachers tend to seek the causes for learning problems out- 
side the classroom and the schools. It is true that a number of external factors 
help explain learning difficulties. However, the fact that the factors found in 
the school and in teacher behaviour are not taken into account may lead to 
actions that only produce frustration and that will hinder future attempts at 
school change. 

® The fifth obstacle has to do with the larger education system. It is a system that 
operates in a vertical top-down manner and that, in general, appears to be dis- 
tant from the school. With a few exceptions, the supervisor does not fulfil 
his/her role adequately and his/her reputation is clearly deteriorated among 
school staff; Nothing is expected from him/her on the part of teachers and 
headmasters. Many schools are in such physical deterioration, or with such an 
evident lack of infrastructure — mainly explained by national economic condi- 
tions that give preference to salaries over infrastructure — that the temptation 
of dedicating the School Project to solve these problems is enormous. The 
request for greater counselling and training, as well as for more frequent vis- 
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its on the part of the technical group, are indications of an education system 
that operates bureaucratically without regard to the school’s needs. It is diffi- 
cult to understand greater autonomy to the schools while this manner of 
operating of the education system remains. 

In spite of all these difficulties, we can end this section on an optimistic note. In one- 
third of the primary schools in the sample one can observe some trends towards 
important change or transformation. Three of these are particularly relevant: 

® Schools begin to think in terms of students learning. School projects in gener- 
al are related to problems directly linked to learning. The central activity of 
the school appears to be revolving around the education of the students. 

® Teachers work collectively and discover the value of teamwork and the value 
of other teachers. In one-third of the schools mentioned frequent meetings of 
the School Technical Council are held during which academic issues are dis- 
cussed because the School Project is part of the agenda and is monitored in 
these meetings. More important, however, are indications that the discovery 
of the value of teamwork seems to be irreversible. A similar trend was observed 
in the few school districts in which periodic meetings of the District Technical 
Council were held. Here the headmasters discovered the value of the inter- 
change of experiences, the role of other principals as counsellors of each other’s 
projects, and the importance of dealing with academic issues. In these cases, 
the supervisor transformed his role and was re-valued by principals. 

® School approaches the community. This is perhaps the most visible change as 
a consequence of the project. Since schools are recommended to consult par- 
ents on the School Project, after which some schools ask for specific 
commitments on their part, both principal and teacher discover that parents 
may become very important allies in the child’s education. Myths in Mexican 
school culture relative to parents — that they are apathetic, uninterested, une- 
ducated — are destroyed. The school becomes nearer to the community and 
takes it more into account. This is another process that might be irreversible. 
There are two ways in which the relationship between the school and the com- 
munity has changed. One refers to the possibility of discussing problems that 
underlie existing conflicts between the school and the community or the par- 
ents. An example is a school that had no fence, exposing the children to the 
risk of accidents in the street. Working together, the school and community 
found the resources to build the fence, and the parents provided the labour to 
build it. This changed the relationships between school and community that 
led to other collaborative projects. Another way is the participation of parents 
in the fostering of their children’s progress in school. In several schools, teach- 
ers were able to state clearly what they expected from the parents: less unjustified 
absenteeism, the reservation of a special time and space for doing homework, 
and the provision of an adequate, nutritious meal before school for the chil- 
dren. The parents became more aware of what was happening in school and 
of their own responsibility towards their children’s progress. 

The changes noted above only occur in one-third of the schools in the sample. 
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Nevertheless, the fact that they are present seems to demonstrate that change that 
stems from the school is possible. On the other hand, they indicate that the trans- 
formation we are after is a slow process that must begin with building the scaffolding 
that will permit an improvement of the quality of the school in the future. This, 
however, must occur at the same time the system is transformed. A small indication 
of this change was seen in two of the school districts we worked in, though this is 
not analysed in this paper. The idea is for the system to position the individual school 
at the centre of its activities and for it to function around its needs and weakness- 
es, as well as its strengths. 

Innovations in assessments^ 

During the previous administration (1994-2000), an important evaluation of stu- 
dent achievement with respect to national standards for primary education in reading 
and mathematics was carried out.*^ Criterion-referenced tests were developed for 
the six grades of primary education in both areas, and were applied to a national- 
ly representative sample of different types of schools: urban private schools, urban 
public schools, rural public schools,’^ Cursos Comunitarios,'^ and Indian bilingual- 
bicultural schools. 

In the following, I refer to main conclusions from the results of the applica- 
tion carried out in May 1998. 

Most Mexican students meet national standards or are near meeting them. 
Nevertheless, about one-third of the students are far from meeting national stan- 
dards, and these grow from the fourth grade onwards. 

However, schools are heterogeneous regarding the achievement of national 
standards in both reading and mathematics. With few exceptions, there are signif- 
icant differences in students’ scores among the different types of schools, which 
reflects the weight of socio-economic and cultural factors on student achievement. 

The exceptions to the previous conclusion, however, are interesting: differences 
in reading tend to disappear after the third grade between rural schools and Cursos 
Comunitarios, and in mathematics, between Cursos Comunitarios and both rural 
and urban public schools. 

There are also important differences in the proportion of students achieving 
standards among the different types of schools. Differences between private and 
public urban schools tend to grow as one progresses in the school grades. The oppo- 
site occurs with the differences between urban and rural public schools, which tend 
to become smaller as one progresses in the school grades. 

An analysis by quartiles and deciles, however, demonstrates the existence of 
strong differences between schools of the same type. The analysis of deciles also 
shows that there are good schools in all populations, from private urban schools to 
indigenous schools,’^ and that a good school is just as good in one population as in 
another. In several cases, good indigenous schools have a greater proportion of stu- 
dents achieving standards than good private schools. This, of course, is a very 
important result, because it demonstrates that excellent schools in any of the cir- 
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cumstances and populations studied exist and, therefore, more can be developed. 

This evaluation based on national standards for primary schools in reading 
and mathematics is considered an innovation for two reasons. The first concerns 
how the tests were constructed: it is possible not only to give an idea of the degree 
of development of basic skills among the primary school students, but also of being 
able to say something about what the students know and what they lack in order 
to meet the standards. For example, in the case of reading, it is clear that in gener- 
al students are able to read sentences and paragraphs, but those that do not meet 
the standards cannot integrate the meaning of a complete text (three or more para- 
graphs, depending on the grade). The second reason concerns the type of analysis 
performed: one of the main objectives is the measurement of inequality in students’ 
learning results between different types of schools. Also, the identification of excep- 
tional schools and of schools presenting severe problems is possible. This evaluation 
was accompanied by contextual information on the child, the family and the school, 
which has not yet been analysed, but which will allow for the hypothetical identi- 
fication of conditions in which the probabilities of having a well-performing school, 
or of having a poorly performing school, become stronger. The relevance of this 
type of analysis for policy making at the different levels is evident. 

Towards a fruitful relationship^^ 

The quality of primary education, as measured by student achievement, is unequal- 
ly distributed among different types of schools in Mexico, In general, the poorer the 
context and the students and their families, and/or the more the children belong to 
families. from cultural (ethnic and linguistic) minorities, the lower the achievement 
of the students and the proportion of students achieving national standards. 

This occurs within a system that functions vertically and that expects schools 
to follow instructions, not to make decisions. However, the evaluation that we have 
described clearly identifies schools in all categories with excellent learning outcomes 
among their students. Excellent schools in the different categories have statistically 
equivalent learning outcomes. It is clear that these schools are the exception and not 
the rule. Nevertheless, it is also obvious that these schools are making their own 
decisions, and probably they are taking them collectively. Only this can explain their 
extraordinary results. 

Thus, decision-making within the school — or a certain and perhaps increasing 
degree of school autonomy — is undoubtedly advisable. 

Nevertheless, in a system in which inequalities in learning outcomes are as 
severe as we have described, autonomy on its own could become a further cause of 
inequality. The capacity of schools for diagnosing their problems, setting collective 
objectives, obtaining clear commitments from the different members of the com- 
munity, and monitoring and evaluating their progress is, as we have seen in the 
second section of this article, also unequally distributed. 

It would, therefore, be a mistake to conclude that the system may be weak- 
ened because schools can be made accountable for their results, as has occurred in 
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England, where local educational authorities were practically done away with and 
the system favoured inspection and evaluation in place of local support. On the con- 
trary, the education system has to become stronger. Aided with results of evaluation 
studies such as the one we have mentioned, the system has to be able to support 
schools according to their capacity to make the right decisions. The education sys- 
tem should^ therefore, use the results of student and school evaluations to identify 
both individual schools and contextual and school conditions that indicate situa- 
tions in need of special pedagogical and managerial support. Interaction between 
schools that have been identified as well performing, and those that have difficul- 
ties in achieving educational standards, could perhaps become one of the strategies 
of support. 

A fruitful relationship among non-punitive assessment, the fostering of school 
autonomy, the differential support to the schools that need it the most on the part 
of the education system, and the requirement of accountability on the part of the 
school, would thus seem the most sensible course for an education system charac- 
terized by inequality such as the Mexican case that we have described here. This, of 
course, requires qualitative leaps on the part of both the schools and the education 
system. Schools must put the student, and his/her learning, at the centre of their pur- 
suit. The system must place each school, according to its conditions, problems and 
potential, at the centre of its activity. 

Notes 

1. I refer to the well-known and growing body of literature on school effectiveness and 
school improvement. 

2. The present administration took office 1 December 2000. For the first time in seventy- 
one years, elections were won by an opposition party. Cf. Rangel Sostmann, 2000. 

3. Secretaria de Educacion Publica. 1997. The course is also offered in the programme for 
the initial training of pre-school and lower secondary schoolteachers. 

4. Secretaria de Educacion Publica. 2000^, The course is also offered for secondary school 
principals and supervisors. 

5. Part of this section is based on Schmelkes, 2001. 

6. There is a programme for the promotion of teachers that includes an evaluation of their 
students. As part of this programme, around 7 million students are tested yearly in pri- 
mary and secondary schools. There is an on-going longitudinal study for the evaluation 
of primary schools that has collected data in the same schools for five years. A diag- 
nostic evaluation of students entering secondary schools (an aptitude test) has been 
carried out for ten years in certain cities within the country. A national evaluation based 
on both primary school standards for reading and mathematics has been carried out 
with a nationally representative sample of schools for three consecutive years, and for 
the first time in secondary schools. Mexico has participated in the following interna- 
tional studies: The TIMSS and the TIMSS-R, the international assessment of third-grade 
students co-ordinated by the Latin American and Caribbean Regional Office of UNESCO 
(Laboratorio Latinoamericano de Evaluacion de la Calidad) and, more recently, the 
PISA project of OECD. 

7. Article 31 of the General Law of Education states: ‘Educational authorities will inform 
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teachers, students, parents, and society at large, of the results of evaluations and of 
other information that allows for the measurement of the development and progress of 
education in each state’. Secretaria de Educacion Publica, 1993. 

8. Cf., for example, Observatorio Ciudadano de la Educacion, 1999. Also in: www.obser- 
vatorio.org, 

9. In Mexico, textbooks for primary school students are free. There are also books for the 
teacher, cards for the development of mathematical competences, and a classroom 
library. It has been found that in many cases these materials are not used, or not suffi- 
ciently exploited. 

10. The project excludes schools that have more than one class per teacher. Many schools 
are ‘multigrade’. It is not easy to have a reliable statistic of these schools, but 25% of 
all primary schools had only one teacher in 1998/99. Ezpeleta 8c Weiss, 2000. 

11. What follows is based on Schmelkes, Forthcoming. 

12. During the last year of the administration, the same exercise was carried out for lower 
secondary education also. However, in this article I refer only to results at the prima- 
ry education level. 

13. There are practically no private rural schools in Mexico. Private primary schools rep- 
resent 5% of the total number of schools in the country, and give service to approximately 
10% of the primary school students. 

14. Cursos Comunitarios is a form of supplying primary education to very small rural com- 
munities (under thirty school-age children). There is one ‘para-teacher’ (secondary school 
graduate) for the three levels (instead of six grades) of primary education. 

15. Indigenous schools offer bilingual education to primary school children of the sixty- 
two different ethnic and linguistic groups in the country. 

16. What follows is based on Secretaria de Educacion Publica, 1999. 
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Historical antecedent 

Learning and civilization have flourished during many eras of Afghan history, A 
cradle and crossroads of ancient civilizations, Afghanistan has contributed to enrich- 
ing the culture of the region in which it is located throughout its history. During the 
Islamic period, many centres of learning were established in cities such as Balkh, 
Herat and Ghazni, which produced scholars in philosophy, literature and science 
to serve the people of the region and beyond. The tenth-century medical treaties 
written by Ibn-i-Seena Balkhi were used in universities until the end of nineteenth 
century. Many Afghan poets and philosophers have enriched human thought and 
civilization. These include the tenth-century Abu Rayhan Biruni, eleventh-century 
Hakim Sanai, thirteenth-century Khwaja Abdullah Ansari and Maulana Jalaludin 
Roumi, and the seventeenth-century Kushal Khan Khattak and Rahman Baba. The 
nineteenth-century Afghan political philosopher Jamaludin Afghani, who had many 
disciples, travelled through Asia, Africa and Europe to promote national indepen- 
dence and Islamic solidarity and to explain the relation between Islam, science and 
progress. 
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At the beginning of the twentieth century, the Islamic tradition permeated every 
aspect of Afghan society, including education and training at all levels. Education 
was provided at home, in the mosques, in informal schools linked to mosques, in 
religious centres (Madrasa) and in circles of scholars. These institutions were sup- 
ported by parents, local communities, religious and tribal leaders and through private 
resources. The Madrasa provided general knowledge and theological studies for 
young men and prepared religious and community guides and teachers. Some oppor- 
tunities for studies in areas such as writing, poetry and literature, history, science 
and traditional medicine were available through private tutoring and in small infor- 
mal circles. Mosques and related schools served as basic community centres in which 
pupils could study the Koran, learn about Islamic values and ethics, and acquire lit- 
eracy and numeracy. Traditional education also helped prepare young people (mainly 
boys) for work through informal apprenticeships in local arts and crafts, farming 
and commerce. 

Educational developments during the twentieth century reflected the religious 
and traditional character of Afghan society. Afghanistan’s changing political con- 
text, including the social and economic policies of the successive regimes and the 
public’s aspiration for the education of their children, influenced the nature and 
form of education and its expansion. In light of the multi-ethnic composition of 
Afghan society, cultural and linguistic policies were important factors in the devel- 
opment of education. Textbooks and educational materials were prepared in two 
national languages (Pashto and Dari). 

A number of factors have hindered educational development in Afghanistan. 
The country is landlocked. Its economy is agricultural (85% of the population live 
in rural areas) and its resources unexploited. Economic constraints have proven to 
be the major obstacles to its educational development. Its GDP per capita was at 
US$60 in 1960, US$160 in 1976, and estimated at about US$600 in 1995. Its domes- 
tic product increased from 12.5 billion Afs. in 1953 to 90.0 billion Afs. in 1976.’ 
In the 1960s, the United Nations classified Afghanistan as one of the world’s twen- 
ty-five least developed countries. 

In the second half of the last century, the international organizations played 
an important role in stimulating educational developments worldwide by setting 
norms and targets and by providing technical and financial assistance. This inter- 
national co-operation contributed to the expansion and qualitative improvement of 
modern education in Afghanistan. 

Pre-Second World War developments 

The beginning of modern education in Afghanistan corresponds with the 1903 estab- 
lishment of the first secondary school, Habibia, in Kabul. Afghan and foreign teachers 
were recruited for Habibia, the aim of which was to train personnel for the civil ser- 
vice. Soon after, several modern primary schools and a teacher training institution 
were established. In 1909, the government set up a Board of Education to supervise 
education across the country, including traditional educational institutions. The 
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board’s duties included that of approving school curriculum and textbooks. Following 
Afghanistan’s independence from the United Kingdom in 1919, King Ammanullah’s 
government made educational development a high priority and the first minister of 
education was appointed in 1922. During the 1920s, a number of primary and sec- 
ondary schools — including a school for girls and an adult education centre for 
women — were established in Kabul, along with several vocational schools in areas 
such as agriculture, arts and crafts and public administration. For the first time, 
Afghan boys and girls were sent abroad for studies. The establishment of diplomatic 
and cultural relations with foreign countries contributed to the development of a 
modern education system. Through bilateral co-operation with countries such as 
Turkey, France and Germany, foreign teachers and expertise as well as student schol- 
arships were made available to Afghans. 

Educational development experienced a set-back in 1929 when modern schools 
were closed during the nine months of civil strife and anarchy that followed the 
January abdication of King Ammanullah. When, following upon this period of con- 
flict, Nadir Shah became king of Afghanistan, the country’s thirteen primary and 
secondary schools were reopened and attention was again given to the development 
of education. A new constitution, adopted in 1931, made primary schooling com- 
pulsory for Afghans and placed all modern educational institutions under the control 
of the State. (Traditional educational entities remained independent.) Education at 
all levels — primary through tertiary — was provided free for Afghans. In 1933, Nadir 
Shah was assassinated and his son Zaher Shah came to the throne. 

The Afghan education system saw some expansion during the 1930s. Special 
attention was given to the development of Pashto as one of the two main national 
languages of Afghanistan. A number of secondary schools were established in the 
provinces and several traditional religious schools were modernized and incorpo- 
rated into the. formal educational structure. A secondary school for girls was 
established in Kabul, along with two secondary vocational schools — one mechani- 
cal, the other agricultural — and a school for the training of assistant doctors and 
pharmacists. Turkish medical advisers and several Afghan doctors, who had trained 
abroad, aided in the creation in 1932 of the Faculty of Medicine — which was later 
to evolve into the University of Kabul, officially established in 1946. The Faculty of 
Medicine graduated the first group of eight intern doctors and twelve pharmacists 
in 1937. With a view to the social and economic development of Afghanistan, in 
1938 the government sent a group of secondary school graduates to universities in 
France, Germany and the United States. 

The pace of educational development in Afghanistan slowed down consider- 
ably during the Second World War — during which Afghanistan remained 
neutral — because of a variety of economic and technical constraints: most foreign 
experts and teachers left the country, and equipment and materials needed for edu- 
cational institutions could not be imported. Nonetheless, by the end of the Second 
World War, the Afghan education system had expanded. During the twenty-year 
period beginning in 1930, the number of students had risen from 1,590 to 95,300, 
4,350 of whom were girls. The number of teachers had increased from fifty-three 
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in 1930 to some 3,000 in 1950; they served 368 schools, including 308 primary 
schools, twenty-five lower secondary schools (middle schools), sixteen upper sec- 
ondary schools (lycees), four vocational schools, one teacher training school and 
seven formal religious schools. There were also 456 students, of which forty were 
female, in five institutions of higher education. By the middle of the twentieth-cen- 
tury, 6% of Afghan children, ages 6-12, were enrolled in primary schools, out of an 
estimated population of 11 million people. 

Post-Second World War developments 

After 1945, in the context of political reform, the beginnings of a short-lived demo- 
cratic movement and the return of a number of Afghan specialists from abroad, 
several important measures were taken to encourage educational development in 
Afghanistan. In 1947, the Ministry of Education was reorganized to ensure the fur- 
ther development of primary, secondary, technical and vocational and teacher 
education. At this time, more focus was also placed on the improvement and pro- 
duction of textbooks, provision of teaching equipment and materials and school 
construction (all of which fell within the ministry’s portfolio). To promote the expan- 
sion of education, increased authority and resources were delegated away from the 
central office and toward provincial offices throughout the country. 

From the early 1950s, the Afghan Government focused on the systematic expan- 
sion of education and the improvement of educational quality. It launched a number 
of development plans, financed through both internal resources as well as bilateral 
and multilateral technical assistance funds and loans from the international com- 
munity — including the United Nations and countries such as the United States, the 
former USSR and Germany. In the first of three five-year development plans (for the 
period 1956-61), 6.5% of the national budget was devoted to social services, with 
958 million Afs. earmarked for education. This plan focused principally on prima- 
ry schooling. 

In view of the country’s need for a trained workforce, the second five-year plan 
(for the period 1962-67) emphasized secondary, technical and higher education. In 
this plan, the share of the national budget for social services was increased to 11.2%, 
with 1,759 million Afs. earmarked for education. The third five-year plan (for the 
period 1968-73) paid special attention to qualitative improvements in teacher edu- 
cation and reflected a commitment to the balanced expansion of education at all 
levels. This time, 16.7% of the total nation budget was allocated to social services, 
with over 3,000 million (or 3 billion) Afs. targeted for education. While primary 
education expanded as a result of these development plans, it was secondary edu- 
cation that experienced the fastest growth — as a result of public demand, especially 
in the provinces. 

Following a coup d’etat in 1973, after which Afghanistan was proclaimed a 
republic and Mohammad Daoud (a former prime minister and cousin of King Zaher) 
became head of state, Afghanistan launched a seven-year economic and social devel- 
opment plan (for the period 1976-83j| accelerate economic growth 
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Basil Bernstein, Karl Mannheim Chair Emeritus in the Sociology of Education, at 
the Institute of Education, University of London, born on 1 November 1924, 
died on 24 September 2000 after a prolonged battle with throat cancer. Professor 
Bernstein was one of the leading sociologists in the world, whose pioneering work 
over the past four decades illuminated our understanding of the relationship among 
political economy, family, language and schooling. Although committed to equity 
and social justice, or in his own words, ‘preventing the wastage of working class 
educational potential’ (19616, p. 308), his work was often misunderstood and 
incorrectly labelled a form of ‘cultural deficit’ theory. Nothing could be more inac- 
curate. 

Raised in London’s East End, the son of a Jewish immigrant family, Bernstein’s 
career reflected his concern for understanding and eliminating the barriers to upward 
social mobility. After serving as an underage bombardier in Africa in the Second 
World War, he worked in the Stepney settlement boys’ club for underprivileged 
Jewish children. He put himself through the London School of Economics by work- 
ing various menial jobs and earned a degree in sociology. He completed teacher 
education at Kingsway Day College and from 1954 to 1960, he taught a variety of 
subjects, including mathematics and physical education, at City Day College in 
Shoreditch. In pure Goffmanesque style, he also taught driver education and motor 
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repair, despite the fact that he did not drive; a fact that he successfully concealed 
from his students. 

In 1960, Bernstein began graduate work at University College, London, where 
he completed his Ph.D. in linguistics. He then moved to the Institute of Education, 
where he stayed for his entire career, rising from senior lecturer to reader to pro- 
fessor, to the Mannheim Chair. During his tenure at the Institute, he also served as 
head of the influential Sociological Research Unit in the 1960s and 1970s and as 
Pro-Director of Research in the 1980s. He continued his prolific writing as an emer- 
itus professor until his death. The recipient of many honorary doctorates and awards, 
he posthumously received the American Sociological Association Sociology of 
Education Section Willard Waller Award for Lifetime Contributions to the sociol- 
ogy of education in 2001. He is survived by his wife of over forty years Marion, a 
psychologist, and their two sons, Saul and Francis. 

The evolution of Bernstein's thought 

For over four decades, Basil Bernstein was an important and controversial sociolo- 
gist, whose work influenced a generation of sociologists of education and linguists. 
From his early works on language, communication codes and schooling, to his later 
works on pedagogic discourse, practice and educational transmissions, Bernstein 
produced a theory of social and educational codes and their effect on social repro- 
duction. Although structuralist in its approach, Bernstein’s sociology drew on the 
essential theoretical orientations in the field — Durkheimian, Weberian, Marxist, and 
interactionist — and provided the possibility of an important synthesis. Primarily, 
however, he viewed his work as most heavily influenced by Durkheim. 

Karabel and Halsey (1977), in their review of the literature on the sociology 
of education, called Bernstein’s work the ‘harbinger of a new synthesis,’ a view 
entirely justified by subsequent events (p. 62). Bernstein’s early sociolinguistic work 
was highly controversial, as it discussed social class differences in language, that 
some labelled a deficit theory. It nonetheless raised crucial questions about the rela- 
tionships among the social division of labour, the family and the school, and explored 
how these relationships affected differences in learning among the social classes. His 
later work (Bernstein, 1977) began the difficult project of connecting power and 
class relations to the educational processes of the school. Whereas class reproduc- 
tion theorists, such as Bowles and Gintis (1976), offered an overtly deterministic 
view of schools without describing or explaining what goes on in schools, Bernstein’s 
work connected the societal, institutional, interactional and intrapsychic levels of 
sociological analysis. 

Bernstein’s early work on language (Bernstein, 1958; 1960; 1961^) examined 
the relationship between public language, authority and shared meanings (Danzig, 
1995, p. 146-47). By 1962, Bernstein began the development of code theory through 
the introduction of the concepts of restricted and elaborated codes (Bernstein, 1962^2; 
1962/?). In the first volume of Class, codes and control (1973^3), Bernstein’s 
sociolinguistic code theory was developed into a social theory examining the rela- 
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tionships between social class, family and the reproduction of meaning systems (code 
refers to the principles regulating meaning systems). For Bernstein, there were social 
class differences in the communication codes of working-class and middle-class chil- 
dren; differences that reflect the class and power relations in the social division of 
labour, family and schools. Based upon empirical research, Bernstein distinguished 
between the restricted code of the working-class and the elaborated code of the mid- 
dle-class. Restricted codes are context dependent and particularistic, whereas 
elaborated codes are context independent and universalistic. 

Although Bernstein’s critics (see Danzig, 1995) argued that his sociolinguistic 
theory represented an example of deficit theory, alleging that he was arguing that 
working-class language was deficient, Bernstein consistently rejected this interpre- 
tation (see Bernstein, 1996, p. 147-56). Bernstein argued that restricted codes are 
not deficient, but rather are functionally related to the social division of labour, 
where context dependent language is necessary in the context of production. Likewise, 
the elaborated code of the middle classes represents functional changes necessitat- 
ed by changes in the division of labour and the middle classes’ new position in 
reproduction, rather than production. That schools require an elaborated code for 
success means that working-class children are disadvantaged by the dominant code 
of schooling, not that their language is deficient. For Bernstein, difference became 
deficit in the context of macro-power relations. 

Beginning with the third volume of Class, codes and control (1977), Bernstein 
developed code theory from its sociolinguistic roots to examine the connection 
between communication codes and pedagogic discourse and practice. In this respect, 
code theory became concerned with the processes of schooling and how they relat- 
ed to social class reproduction. Bernstein’s quest for understanding the processes of 
schooling led him to continue to pursue the fruitful avenue of inquiry developed in 
his article ‘Class and pedagogies: visible and invisible’ (Bernstein, 1977, p. 116-56). 
In that article, Bernstein analyzed the differences between two types of educational 
transmission and suggested that the differences in the classification and framing rules 
of each pedagogic practice (visible = strong classification and strong framing; invis- 
ible = weak classification and weak framing) relate to the social-class position and 
assumptions of the families served by the schools. (For a detailed analysis of this 
aspect of Bernstein’s work, see Atkinson, 1985; Atkinson, Davies & Delamont, 
1995; Sadovnik, 1991; 1995.) The article clearly demonstrated that sociologists of 
education had to do the difficult empirical work of looking into the world of schools 
and of linking educational practices to the larger institutional, societal and histori- 
cal factors of which they are a part. 

The concept of classification is at the heart of Bernstein’s theory of pedagogic 
discourse and practice. Classification refers to ‘the degree of boundary maintenance 
between contents’ (Bernstein 1973a, p. 205; 19736, p. 88) and is concerned with 
the insulation or boundaries between curricular categories (areas of knowledge and 
subjects). Strong classification refers to a curriculum that is highly differentiated and 
separated into traditional subjects; weak classification refers to a curriculum that is 
integrated and in which the boundaries betweenji^’^ts are fragile. 
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Using the concept of classification, Bernstein outlined two types of curriculum 
codes: collection and integrated codes. The first refers to a strongly classified cur- 
riculum; the latter, to a weakly classified curriculum. In keeping with his Durkheimian 
project, Bernstein analyzed the way in which the shift from collection to integrated 
curriculum codes represents the evolution from mechanical to organic solidarity (or 
from traditional to modern society), with curricular change marking the movement 
from the sacred to the profane. 

Whereas classification is concerned with the organization of knowledge into 
curriculum, framing is related to the transmission of knowledge through pedagog- 
ic practices. Framing refers to the location of control over the rules of communication 
and, according to Bernstein (1990), ‘if classification regulates the voice of a cate- 
gory then framing regulates the form of its legitimate message’ (p. 100). Furthermore, 
‘frame refers to the degree of control teacher and pupil possess over the selection, 
organization, pacing and timing of the knowledge transmitted and received in the 
pedagogical relationship’ (1973fc, p. 88). Therefore, strong framing refers to limit- 
ed options between teacher and students; weak framing implies more freedom. 

Bernstein developed this approach into a systematic analysis of pedagogic dis- 
course and practices. First, he outlined a theory of pedagogic rules that examined 
the ‘intrinsic features which constitute and distinguish the specialized form of com- 
munication realized by the pedagogic discourse of education’ (Bernstein, 1990, 
p. 165). Second, he related his theory of pedagogic discourse to a social-class base 
and applied it to the development of educational practices (Bernstein, 1990, p. 63-93). 

The concept of code was central to Bernstein’s sociology. From the outset of 
its use in his work on language (restricted and elaborated codes), code refers to a 
‘regulative principle which underlies various message systems, especially curriculum 
and pedagogy’ (Atkinson, 1985, p. 136). Curriculum and pedagogy are considered 
message systems, and with a third system, evaluation, they constitute the structure 
and processes of school knowledge, transmission and practice. As Bernstein (1973fo) 
noted: ‘Curriculum defines what counts as valid knowledge, pedagogy defines what 
counts as valid transmission of knowledge, and evaluation defines what counts as 
a valid realization of the knowledge on the part of the taught’ (p. 85). Thus, his the- 
ory of education must be understood in terms of the concepts of classification, 
framing and evaluation, and their relationship to his sociological project. 

Following this earlier work on curriculum and pedagogic practice was a detailed 
analysis of pedagogic discourse that presented a complex analysis of the recontex- 
tualization of knowledge through the pedagogic device (see Bernstein, 1990). 
Bernstein’s work on pedagogic discourse was concerned with the production, dis- 
tribution and reproduction of official knowledge and how this knowledge is related 
to structurally determined power relations. What is critical is that Bernstein was 
concerned with more than the description of the production and transmission of 
knowledge; he was concerned with its consequences for different groups. 

Bernstein’s analysis of pedagogic practice looked at the process and content of 
what occurs inside schools. His theory of pedagogic practice examined a series of 
rules, considered both how these rules affect the content to be transmitted and, per- 

152 

Prospects, voL XXX/, no. 4, December 2001 




Basil Bernstein 



611 



haps more important, how they ‘act selectively on those who can successfully acquire 
it/ From an analysis of these rules, Bernstein examined ‘the social class assumptions 
and consequences of forms of pedagogic practice’ (Bernstein, 1990, p. 63). Finally, 
he applied this theory to conservative/traditional versus progressive/child-centred 
practices. He differentiated between a pedagogic practice that is dependent on the 
economic market — that emphasizes vocational education — and another that is inde- 
pendent and autonomous of the market — that is legitimated by the autonomy of 
knowledge. Bernstein concluded that both, despite their claims to the contrary, would 
not eliminate the reproduction of class inequalities. Through a consideration of the 
inner workings of the types of educational practice, Bernstein contributed to a greater 
understanding of how schools reproduce social-class advantages in society. 

Bernstein’s analysis of the social-class assumptions of pedagogic discourse and 
practice is the foundation for linking micro-educational processes to the macro-soci- 
ological levels of social structure and class and power relations. His thesis was that 
there are significant differences in the social-class assumptions of visible and invis- 
ible pedagogy and despite these differences there may indeed be similar outcomes, 
especially in the reproduction of power and symbolic control. 

Thus, from his early work on code theory to the more recent works in Class, 
codes and control, volumes 4 and 5 on pedagogic discourse (1990, p. 165-218), 
and on pedagogic practices (1990; 1996), Bernstein’s project sought to link micro- 
processes (language, transmission and pedagogy) to macroforms — to how cultural 
and educational codes and the content and process of education are related to social 
class and power relations. 

Bernstein and sociological theory 

Karabel and Halsey argued that one of the most unresolved problems of Bernstein’s 
work is how ‘power relationships penetrate the organization, distribution and eval- 
uation of knowledge through the social context’ (qtd. in Karabel & Halsey, 1977, 
p. 71). From the 1970s on, Bernstein continued to search for answers to this ques- 
tion and developed an increasingly sophisticated model for understanding how the 
classification and framing rules of education affect the transmission, the distribu- 
tion and, perhaps, the transformation of consciousness, and how these processes 
are indirectly related to the economic field of production. 

Bernstein conceded that those who seek answers to difficult educational ques- 
tions often prefer a top-down approach — one that begins with the large policy 
questions and builds down to an analysis of how the schools work to provide solu- 
tions or to constrain their formulation. He admitted, however, that the nature of 
his project was to build from the bottom to the top — an approach that sought to 
write the rules of educational process; then to link them to larger structural condi- 
tions; and, finally, to place this analysis in the context of the larger educational and 
policy questions of educators (Bernstein, 1990). 

His theoretical approach has been labelled Durkheimian, neo-Marxist, struc- 
turalist and interactionist, as well as being part of the ‘new sociology’. Bernstein 
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(1996) stated that these have been the labels of others and that they have often been 
too exclusive, often simplifying the theoretical complexity of his model. He acknowl- 
edged that Durkheim has always been at the heart of his sociological theory, in part 
as a corrective to the conservative interpretation of Durkheim’s work, especially in 
the United States; in part as a consequence of Parson’s structural-functional inter- 
pretation of Durkheim. Additionally, although he acknowledged the structuralist 
interpretations of his work by Atkinson (1985) and Sadovnik (1991), he did not 
see his work as exclusively structuralist. He rejected the view that he was part of 
the ‘new sociology’, as he believed that his work was ‘old’ sociology, particularly 
in terms of its roots in classical sociological theory. Finally, he suggested that the 
idea that it was his project to connect disparate sociological theories was not his 
but was suggested by others, particularly Karabel and Halsey (1977). Although 
their labelling of his work as the ‘harbinger of a new synthesis’ was complimenta- 
ry, it also raised an expectation of a kind of synthesis that has not been explicitly 
part of his project. Rather than working from one sociological theory, or attempt- 
ing to synthesize a number of theories, Bernstein attempted to develop and refine a 
model that is capable of describing the different aspects of society. 

Bernstein’s project, from his early work on language, to the development of 
code theory, to the work on curriculum and pedagogic practice and discourse, was 
to develop a systematic theory that provides an analytic description of the way in 
which the education system is related to the social division of labour. His work had 
at its core the goal of his entire project: to develop a Durkheimian theory that ana- 
lyzed the way in which changes in the division of labour create different meaning 
systems and codes, provided analytic classifications of these systems, and incorpo- 
rated a conflict model of unequal power relations into its structural approach. 

Atkinson (1981; 1985) argued that the evolution of Bernstein’s sociology must 
be understood as the movement from its early Durkheimian roots to a later conver- 
gence with European structuralist thought, especially French social theory. In the 
United States, however, because the Durkheimian tradition was appropriated both 
by Parsonian structural-functionalism and by positivism, Bernstein’s work was rarely 
linked to Durkheim and structuralism or was criticized for being linked to Durkheim. 
For example, Karabel and Halsey (1977) spoke of his need to link his Durkheimian 
perspective more explicitly to neo-Marxist categories. While his work on pedagog- 
ic discourse and practice clearly did link the two, Bernstein never moved out of a 
Durkheimian position; rather, he incorporated the neo-Marxist and Weberian cat- 
egories of class and power relations into his overall theory. It is necessary to remove 
the consensus aspects of functionalism that are associated with structural-function- 
alism to understand Bernstein’s sociology. Although his work has been concerned 
with how communication, cultural and educational codes function in relation to 
social structures, Bernstein was concerned not with the way in which such func- 
tioning leads to consensus but with how it forms the basis of privilege and domination. 

It is with respect to the relationship with privilege and domination that 
Bernstein’s work, while remaining consistent with a Durkheimian foundation, sys- 
tematically integrated Marxism and Weberian categories and provided the possibilities 
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for the synthesis for which Karabel and Halsey call. Bernstein’s work continued to 
be Durkheimian because, as Atkinson (1985, p. 36) pointed out, an essential activ- 
ity has been the exploration of changes from mechanical to organic solidarity through 
an analysis of the division of labour, boundary maintenance, social roles, the ritu- 
al-expressive order, cultural categories, social control and types of messages. It 
attempted to look at modes of cultural transmission through the analysis of codes. 
In addition, his work continued to link classification and framing codes to the unequal 
distribution of resources in capitalist societies. While the early work on class and 
pedagogy was clearly more Durkheimian in its analysis of changes in organic soli- 
darity, his later work (Bernstein, 1990; 1996) was more interested in the consequences 
of different pedagogic practices for different social classes and, most important, 
returned to the very questions of education and inequality that were the original 
basis of the project over forty years ago. 

Thus, Bernstein’s project, since the 1970s, accomplished a number of related 
and important things. First, it provided a theory of school knowledge and trans- 
mission and demonstrated how the what of education is transmitted. Second, it 
linked the sociolinguistic aspects of his early work to the analysis of the codes of 
schooling. Third, in relating the process and content of schooling to differences in 
social class and in calling for an analysis of the consequences of those differences in 
curriculum and pedagogy, Bernstein provided a tentative integration of structural- 
ist and conflict approaches within sociology. 

Criticism of Bernstein's work 

Much of the criticism of Bernstein’s early work revolved around issues of deficit and 
difference. Bernstein rejected the view that his work was based on either a deficit or 
a difference approach. Rather, he argued that his code theory attempted to connect 
the macro-levels of family and educational structures and processes and to provide 
an explanation for unequal educational performance. He stated: 

The code theory asserts that there is a social class regulated unequal distribution of privileg- 
ing principles of communication [. . .] and that social class, indirectly, effects the classification 
and framing of the elaborated code transmitted by the school so as to facilitate and perpet- 
uate its unequal acquisition. Thus the code theory accepts neither a deficit nor a difference 
position but draws attention to the relations between macro power relations and micro prac- 
tices of transmission, acquisition and evaluation and the positioning and oppositioning to 
which these practices give rise (1990, p. 118-19). 

Despite Bernstein’s continued refutation of the cultural deprivation label, these dis- 
tortions had profoundly negative consequences for his work. For example, Hymes 
reported: ‘a young anthropologist recently told me that as a student she found 
Bernstein’s account of restricted code to describe her own family but was told by a 
faculty member not to read him’ (Hymes, 1995, p. 5). When Bernstein came to a 
United States university in 1987, an anthropologist asked why ‘that fascist Bernstein 
[had been] invited’. When pressed, the anthropologist admitted that she had never 
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read Bernstein’s own work, but that she had read secondary sources accusing him 
of racism. Danzig cites examples in textbooks written in the 1990s that continue to 
portray Bernstein in this light (Danzig, 1995, p. 152). 

The mischaracterization of Bernstein’s work in the 1960s and 1970s contin- 
ued to affect Bernstein’s standing in the intellectual field through the 1990s. Although 
the Bernstein symposium at the American Educational Research Association’s (AERA) 
annual meeting in 1991, Atkinson and Sadovnik’s 1995 volumes, and Bernstein’s 
appearance at AERA’s 1996 annual meeting did much to refute these negative claims, 
significant damage had already been done. 

A second criticism regards Bernstein’s writing style, which many found dense, 
difficult and often incomprehensible (Walford, 1995, p. 193). Although Bernstein’s 
work was indeed complex and difficult, this is no less true of other major sociolog- 
ical theorists, most notably Pierre Bourdieu (Swartz, 1997). In fact, it is in comparison 
to Bourdieu that some critics found Bernstein’s work wanting. 

Marker and May (1993) indicated that despite overlapping concerns, Bourdieu 
provided a more flexible approach to the structure/agency problem in social theo- 
ry. Through a comparison of Bernstein’s concept of code and Bourdieu’s concept 
of habitus, the authors argued that Bernstein was a structuralist, a position that they 
believe Bourdieu had rejected, and that Bernstein’s concept of code resulted in an 
overemphasis on ‘rules’. Bourdieu’s concept of habitus, they argued, resulted in the 
more flexible idea of ‘strategy’, which Marker and May suggested resulted in a less 
dichotomous view of structure and agency (1993, p. 169). Bernstein (1996) respond- 
ed to the Marker and May thesis by saying that it was one more example of 
‘misrecognition’. He accused the authors of recycling out-of-date definitions of code 
and misreading code theory (p. 182-201). Through a detailed use of various quo- 
tations from his work over time, Bernstein rejected Marker and May’s criticism that 
his structuralism denied human agency. 

Marker and May’s article also revealed significant disagreements between 
Bernstein and Bourdieu. For example, they quoted Bourdieu on Bernstein: 

To reproduce in scholarly discourse the fetishing of legitimate language which actually takes 
place in society one has only to follow the example of Basil Bernstein who describes the prop- 
erties of the elaborated code without relating this social product to the social conditions of 
its production and reproduction or even as one might expect from the sociology of educa- 
tion to Its own academic conditions (Bourdieu, 1991, p. 53), 

Bernstein, responding to this Bourdieu quote stated, ‘This comment, reproduced 
with evident approval by Marker and May, is not simply inaccurate, or only sloven- 
ly scholarship, but bizarre. If it reveals anything it reveals the activities of the 
intellectual field, its positioning, position taking and strategies in a somewhat prim- 
itive mode’ (Bernstein, 1996, p. 183). 

Bernstein, too, was critical of Bourdieu. He distinguished code from habitus 
in the following way: ‘The concept of code bears some relation to Bourdieu’s con- 
cept of habitus. The concept of habitus, however, is a more general concept, more 
extensive and exhaustive in its regulation. It is essentially a cultural grammar spe- 
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cialized by class positions and fields of practice’ (Bernstein, 1990, p. 3). Bernstein 
went on to argue that theories like Bourdieu’s were concerned with understanding 
‘how external power relations [were] carried by the system . . . [and not] with the 
designation of the carrier, only with a diagnosis of its pathology’ (1990, p. 172). 

Another criticism of Bernstein’s work has been that it lacked empirical testing 
and support. King (1976; 1981) tested Bernstein’s early model of pedagogic prac- 
tice but did not find strong evidence in his research to support this model; however, 
Tyler (1984) argued that King’s statistical methods were severely flawed. More 
recently, researchers (see Sadovnik, 1995, Parts IV and V; Morals et al., 2001) have 
provided empirical evidence to support Bernstein’s work. A more detailed account 
is provided in the next section. 

Whatever the criticisms of his work, it is undeniable that Bernstein’s work rep- 
resents one of the most sustained and powerful attempts to investigate significant 
issues in the sociology of education. Forty years ago, Bernstein began with a simple 
but overwhelming issue: how to find ways to ‘prevent the wastage of working-class 
educational potential’ (Bernstein, 1961fo, p. 308). The problem of educability led to 
the development of code theory. Code theory, while a powerful and controversial 
perspective on educational inequality, did not sufficiently provide an understand- 
ing of what goes on inside the schools and how these practices are systematically 
related to social-class advantages and disadvantages. In an attempt to connect the 
macro and the micro further, Bernstein’s work since the 1960s centred on a model 
of pedagogic discourse and practices, beginning with the concepts of classification 
and framing and continuing to a more systematic outline of the ‘what’ and the ‘how’ 
of education. Taken as a whole, Bernstein’s work provided a systematic analysis of 
codes, pedagogic discourse and practice and their relationship to symbolic control 
and identity. 

Bernstein's influence on educational research 

Bernstein had a profound influence on sociological research on education. He point- 
ed to years of empirical research in the United Kingdom and elsewhere, which 
attempted to test his theories. Studies conducted by his doctoral students at the Uni- 
versity of London’s Institute of Education and others have contributed to our 
knowledge of the relationships between the division of labour, the family and school- 
ing through research on specific aspects of Bernstein’s work. In a detailed and 
comprehensive chapter in his last book, Pedagogy, symbolic control and identity 
(1996), Bernstein provided a historical discussion of code theory and outlined some 
of the empirical work to test it. As rhe research in the 1960s and early 1970s was 
often conducted by Bernstein’s Ph.D. students as their dissertation research, the 
Sociological Research Unit at the Institute of Education, which he directed, became 
a primary testing ground for Bernstein’s theories. 

The core of Bernstein’s early work was to develop a code theory that exam- 
ined the interrelationships between social class, family and school. By 1971, Bernstein 
had developed an Index of Communication and Control to measure different fam- 
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ily types and to relate them to social class differences. As the original index, accord- 
ing to Bernstein (1996, p. 96), was crude and indirect, Bernstein sought to develop 
a more direct and sensitive measure. Based upon empirical research, Bernstein and 
Jenny Cook-Gumperz developed ‘complex principles of descriptions of the speech 
of parents and children’ (Bernstein & Cook-Gumperz, 1973). Cook-Gumperz pro- 
vided an in-depth description of these principles in her own work (Cook-Gumperz, 
1973, p. 48-73). 

In the 1970s, a number of empirical studies examined the concepts of classifi- 
cation and framing. Neves (1991) studied the relationship between the pedagogic 
codes of families and schools and provided empirical support for Bernstein’s thesis. 
Ana Marie Morais and her colleagues (Morais, Peneda, Madeiros, 1991; Morais et 
al., 1991) demonstrated that it was possible to design different pedagogic practices 
and to evaluate their outcomes. She designed three different pedagogic practices in 
terms of varying degrees of classification and framing and trained a teacher to teach 
the same subject to four different classes using different pedagogic practices. Based 
upon her research the complex relationship between the pedagogic code of the fam- 
ily and the school, social class differences in families, the educational development, 
the educational achievement and behaviour of the child was more fully understood. 

Bernstein’s analysis of the relationship between social class and pedagogic prac- 
tice was confirmed by Jenkins’ research (1990) on the social class basis of progressive 
education in Britain. Through an analysis of articles in the Journal of the New 
Education Fellowship between 1920 and 1950, she supported Bernstein’s central 
thesis about the social class basis of invisible pedagogy, which Jenkins argued was 
precisely what the progressives were talking about. Semel (1995) further supported 
this thesis as applied to independent schools in the United States from 1914 to 1935. 

The relationship between the fields of symbolic control and production and 
gender classification was explored by Holland (1986). Her study concluded that 
socialization processes differ in classification and framing in relation to the place of 
families in the division of labour. Families in the field of symbolic control have weak- 
er classification in their modelling of domestic and economic divisions of labour 
than families in the field of production. Holland’s work provided important empir- 
ical evidence to support Bernstein’s thesis that classification and framing are social 
class related and related to the fields of production and symbolic control. Further, 
this study broadened the emphasis away from class reproduction to the related and 
equally significant area of gender role reproduction. 

The work of Diaz (1984; 1990) and Cox Donoso (1986) examined Bernstein’s 
theory of pedagogic discourse. Diaz’s research explored the institutionalizing of pri- 
mary education as a form of pedagogic discourse. Cox Donoso work on State 
education in Chile related the model of pedagogic discourse to the field of symbol- 
ic control. Cox Donoso’s research compared the educational policies of the Christian 
Democratic Party and Allende’s Popular Unity Party. Through an analysis of the 
relationship between pedagogic discourses and each party’s relationship to the sym- 
bolic and economic fields, Cox Donoso provided a concrete sociological and historical 
testing of Bernstein’s theory. 
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Although much of the research on his theory has been produced by his own 
Ph.D. students, there were numerous other studies using his work. By 1996, Bernstein 
reported fifteen articles in the British journal of the sociology of education based 
on the theory. Two published collections (Atkinson, Davies & Delamont, 1995; 
Sadovnik, 1995) provided numerous examples of how Bernstein’s work influenced 
an international group of educational researchers. Most recently, Mprais, Neves, 
Davies and Daniels (Morais et ah, 2001) have edited a collection of articles on 
Bernstein’s contributions to research, which were presented at a symposium 
in Lisbon in June 2000. Among the research based on Bernstein’s work are inves- 
tigations of pedagogic discourse by Parlo Singh and Karen Dooley, Johann 
Muller, Rob Moore and Karl Maton, and Mario Diaz; on sociolinguistics by 
Ruqaiya Hasan, and Geoff Williams, and on technology by William Tyler, and by 
Bernstein himself. Additionally, Madeleine Arnot (2001) has written on how 
Bernstein’s work affected feminist educational researchers and theorists. 

What is clear is that over a forty-year period, Bernstein developed a systemat- 
ic code theory, which was constantly refined and developed and which, through his 
students and other researchers, has been empirically researched. Moreover, Bernstein’s 
theories underwent revision and clarification in light of this research. What comes 
through in his own reflections on his sociological project is how theory and research 
were crucially related to each other. 

Afterward: Basil Bernstein; mentor and friend 

I first met Basil Bernstein in 1978 at New York University, when I was a doctoral 
student and he was a visiting professor. He took an interest in a paper I had writ- 
ten for him applying his work to Bowles and Gintis’ Schooling in capitalist America. 
For the next years, he was my mentor, colleague and, most of all, beloved friend. 

As a mentor, he was giving of his time and support. Although he responded 
favourably to my work on him, he nonetheless responded with long letters, always 
hand-written, always difficult to decipher, pointing to things I had overlooked, new 
ways of seeing, and full of new insights. While some warned that writing about his 
work could damage our friendship, it never did. Even when he disagreed with my 
interpretations, he never asked that I change a word. The process of editing Knowledge 
and pedagogy was one of the most intense and satisfying experiences of my career. 
Bernstein read and wrote responses to many of the articles in the book; his corre- 
spondence on the book is filled with incredible contributions to my own thinking, 
only a portion of it included in his epilogue. Most of all, Bernstein never forgot that 
it was my book, not his, and after providing feedback, left the final editing to me. 
For the next twenty-two years, what began with my watching his incredible mind 
work out models from the third volume of Class, codes and control at New York 
University, continued as I moved from doctoral student to professor: Bernstein helped 
me understand the complexities of schooling and social reproduction. As a teacher, 
he inspired me to help my own students grow and develop intellectually; as a schol- 
ar, he inspired me to think sociologically and to insist upon empirical research to 
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support theory. 

What I will always cherish most is Basil Bernstein’s friendship. I will always 
remember the wonderful times we shared with him and his wife Marion (to whom 
he was devoted) at their lovely home in Dulwich, at the National Theatre, at the 
Tate Gallery, shopping at Harvey Nichols, Liberty’s and on Bond Street, and eat- 
ing and drinking in numerous restaurants near the Institute in Bloomsbury. Bernstein 
was no narrow academic. He was an arts aficionado, most proud of his David 
Hockneys; an audiophile, who moved reluctantly from his precious LP collection 
to CDs; an expert photographer, who was as proud of his photo of Susan Semel in 
the Hofstra University Research Magazine, complete with the credit, ‘photograph 
by Basil Bernstein’, as he was of a journal article; a Beau Brummel, he was fond of 
Armani and Kenzo. And what a conversationalist he was: ironic, creative, clever, 
amusing, knowledgeable, and at times, cryptic and sardonic. Whether it was apply- 
ing code theory to the exploitation of South American farmers at one of his favourite 
Bloomsbury haunts, Isolabella, or with Eliot Freidson, entertaining us with their 
tales of 1968 at Berkeley, Bernstein was one of a kind. 

The last time I saw Basil Bernstein was in June 2000, upon journeying to London 
from a conference in Lisbon organized by Ana Morais, Isabel Neves, Harry Daniels 
and Brian Davies on his contributions to educational research. Too ill to attend as 
planned, he participated on Friday for the last hour via video link to his home in 
London. Despite being weak from treatment, he was vintage Basil Bernstein: witty, 
creative, and dressed for the occasion in one of his favourite silk shirts. His brief 
written contribution on code theory and technology provided significant food for 
thought. Upon termination of the link, there was not a dry eye among us. We all 
knew that this might have been his last public appearance and we all knew how 
much we would miss him. 

On Sunday, following the conference, Susan Semel and I visited Basil Bernstein 
and his wife Marion in London. Although weak, he spoke of finishing the sixth vol- 
ume of Class, codes and control, applying code theory to the Internet and technology, 
and of New Labour educational policy, still in his view, like Thatcher’s, ‘a new ped- 
agogical Janus [. . .]’{Bernstein, 1990) reproducing the old inequalities. Although I 
left hoping it was not a final goodbye, I knew that it might well be. And it was. 
When Basil Bernstein died on 24 September 2000, the world of sociology lost a 
giant. I lost a mentor and friend to whom I will always be grateful. 

Note 

1. Parts of this article are adapted from Sadovnik, 1991 and 1995 (see references). 
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